ClickHouse disk and memory pressure on self-hosted LangSmith

Symptom

High-throughput self-hosted LangSmith deployments hit ClickHouse OOM-kills and disk pressure even when the LangSmith application tables (runs, feedbacks, etc.) are small. Disk usage is dominated by system.*_log tables and by orphaned copies of those tables with numeric suffixes (_0, _1).

Cause

Two things compound:

• No default TTL on ClickHouse system logs. system.query_log, system.trace_log, system.part_log, system.query_views_log, and system.metric_log grow forever unless you configure TTLs. The LangSmith Helm chart does not set these. Deployments running for months accumulate many gigabytes.

• Orphaned migration tables. When ClickHouse migrates a system table's schema, it renames the old table with a numeric suffix (system.query_log_0, system.query_log_1, etc.) and creates a fresh canonical table. The renamed copies are never dropped and never touched again. Across upgrade cycles they add up.

Resolution

All four steps below are confirmed safe by LangChain engineering. Steps 1, 2, and 4 address disk/memory directly. Step 3 is optional but removes a heavy writer.

1. Find and drop orphaned suffix tables

List them first:

SELECT database, name, formatReadableSize(total_bytes) AS size
FROM system.tables
WHERE database = 'system'
  AND match(name, '.*_[0-9]+$')
ORDER BY total_bytes DESC;

Then drop. LangSmith never reads the suffixed variants.

SET lock_acquire_timeout = 5;
DROP TABLE IF EXISTS system.query_log_0;
DROP TABLE IF EXISTS system.query_log_1;
DROP TABLE IF EXISTS system.trace_log_0;
DROP TABLE IF EXISTS system.trace_log_1;
DROP TABLE IF EXISTS system.part_log_0;
DROP TABLE IF EXISTS system.part_log_1;
DROP TABLE IF EXISTS system.metric_log_0;
DROP TABLE IF EXISTS system.metric_log_1;
-- Repeat for any other _N variants you found above

lock_acquire_timeout = 5 stops the DROPs from hanging if another operation holds a lock. Bump it if needed.

2. Set TTLs on system log tables via Helm

Inject an XML drop-in under /etc/clickhouse-server/config.d/ using a ConfigMap and a volumeMount on the ClickHouse StatefulSet.

ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: langsmith-clickhouse-system-logs-ttl
  namespace: langsmith
data:
  system_logs_ttl.xml: |
    <clickhouse>
      <query_log>
        <ttl>event_date + INTERVAL 30 DAY</ttl>
      </query_log>
      <trace_log>
        <ttl>event_date + INTERVAL 30 DAY</ttl>
      </trace_log>
      <part_log>
        <ttl>event_date + INTERVAL 30 DAY</ttl>
      </part_log>
      <query_views_log>
        <ttl>event_date + INTERVAL 30 DAY</ttl>
      </query_views_log>
    </clickhouse>

Apply it: kubectl apply -f clickhouse-system-logs-ttl-configmap.yaml

Helm values:

clickhouse:
  statefulSet:
    volumes:
      - name: system-logs-ttl-config
        configMap:
          name: langsmith-clickhouse-system-logs-ttl
    volumeMounts:
      - name: system-logs-ttl-config
        mountPath: /etc/clickhouse-server/config.d/system_logs_ttl.xml
        subPath: system_logs_ttl.xml

Upgrade the release:

helm upgrade langsmith langchain/langsmith -f values.yaml -n langsmith

TTLs take effect after the ClickHouse pod restarts. Cleanup happens on background merges. To force it:

OPTIMIZE TABLE system.query_log FINAL;
OPTIMIZE TABLE system.trace_log FINAL;
-- etc.

OPTIMIZE ... FINAL is heavy on large tables; run it in a low-traffic window. 30 days is an example; minimum recommended is 7 days.

3. Disable the C++ query profiler (optional)

The profiler writes stack samples into system.trace_log at a high rate. LangSmith does not read that table, so you can turn it off. Inject via the same ConfigMap/volumeMount pattern:

<clickhouse>
  <profiles>
    <default>
      <query_profiler_real_interval_ns>0</query_profiler_real_interval_ns>
      <query_profiler_cpu_interval_ns>0</query_profiler_cpu_interval_ns>
    </default>
  </profiles>
</clickhouse>

Setting both to 0 disables CPU and wall-clock sampling. No LangSmith metrics, alerts, or dashboards depend on this.

4. Apply TTLs on LangSmith application tables

LangSmith tolerates missing secondary records (no 500s if history or feedback rows have been pruned). Confirm the date column with DESCRIBE TABLE <table> before running.

ALTER TABLE default.runs_history   MODIFY TTL toDate(start_time) + INTERVAL 30 DAY;
ALTER TABLE default.feedbacks      MODIFY TTL toDate(created_at) + INTERVAL 30 DAY;
ALTER TABLE default.billable_traces MODIFY TTL toDate(start_time) + INTERVAL 90 DAY;

billable_traces can safely use 90 days: LangSmith billing is aggregated externally and does not depend on historical rows in that table.

Safety matrix

Checking table sizes

SELECT database, name, formatReadableSize(total_bytes) AS size, total_rows
FROM system.tables
WHERE database IN ('system', 'default')
ORDER BY total_bytes DESC
LIMIT 30;

References

• ClickHouse system.query_log and TTL

• ClickHouse MergeTree TTL expressions

• ClickHouse config.d drop-in configuration

• LangSmith self-hosted Helm chart