LangSmith bulk export OOM: tuning buffer size and concurrency

Last updated: July 7, 2026

Symptom

On self-hosted LangSmith, a large daily bulk export repeatedly fails with this pattern:

  1. The export starts and may complete an initial batch of partitions.

  2. When running the export the langsmith-queue pod is OOM-killed by the kernel. Especially during peak hours, or when traces carry large inputs, outputs, or extra payloads.

  3. In-flight partitions cannot transition to errored and stay stuck in Running.

  4. The replacement pod sees the max concurrent runs already Running (e.g. {Completed: 25, Created: 250, Running: 5}) and picks up no new work.

  5. The export is frozen until the stuck partitions auto-clear (~48 hours) or are manually cleared.

Solutions like raising pod memory alone (e.g. 4Gi or 8 Gi) does not reliably prevent the OOM. Scaling to multiple pods pods at higher memory may get most partitions through, but it is expensive and does always resolve the issue.

Relevant version: 0.15.x

Cause

Each bulk export partition pulls rows from ClickHouse, accumulates them in an in-memory buffer until the buffer hits DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB, then serializes the whole buffer to Parquet in one shot and uploads it. The serialize-all-at-once step produces a transient memory spike larger than the steady buffer size.

In addition, up to BULK_EXPORT_MAX_CONCURRENT_RUNS partitions run concurrently inside the same langsmith-queue process.

The default values for these are:

  • DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB = 100000 (100 MB buffer ceiling per partition)

  • BULK_EXPORT_MAX_CONCURRENT_RUNS = 5 (5 partitions at once, globally)

A single pod can therefore hold 5 × 100 MB buffers plus 5 concurrent serialization spikes. There's also the original data for those buffers, the parquet/serialization buffers, and other data which multiplies the memory load, as well as other queue jobs.

With large peak-hour payloads this pushes well past 8 Gi and trips the OOM killer.

Because the kill is a forced kernel signal, the running partitions cannot mark themselves errored. The auto-recovery timeout for stuck Running partitions is about 48 hours, which is too slow for a daily export.

Settings live in lc_config/lc_config/settings.py (the bulk export block). Buffer logic is in smith-backend/app/models/bulk_exports/export.py; concurrency and orchestration in smith-backend/app/models/bulk_exports/jobs.py.

Resolution

To relieve the memory pressure, we can finetune the two environment variables below on the langsmith-queue deployment (via Helm values or the deployment spec). Both are read at process start.

1. DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB

  • Default: 100000 (100 MB)

  • Recommended starting value: 30000 (30 MB)

  • Effect: partitions flush and upload sooner, cutting per-partition peak memory by ~70%. This is the single highest-impact change.

  • Trade-off: more frequent flush/upload cycles per partition.

2. BULK_EXPORT_MAX_CONCURRENT_RUNS

  • Default: 5

  • Recommended direction: lower it. A good starting heuristic is to set it to the number of langsmith-queue pods, so each pod averages one concurrent run. You can add more pods to run at a smaller concurrency per-pod, but higher number overall, to speed up the export.

  • Effect: fewer partitions buffer and serialize inside the queue process at once. Combines multiplicatively with a lower buffer ceiling.

Trim exported fields

If BigQuery does not need the full trace payload, drop or trim large fields (inputs, outputs, extra) from the export schema. Peak-hour OOMs correlate with large payload traces, so this directly shrinks the buffer.

Recommended tuning process

  1. Set DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB: 30000 on langsmith-queue.

  2. Lower BULK_EXPORT_MAX_CONCURRENT_RUNS. It defaults at 5, and if it works together with max batch payload size, leave it there, but if needed, you can lower this number.

  3. Drop or trim unused large fields from the export schema if possible.

  4. Run the next scheduled export and watch pod memory and export progress.

  5. Adjust both variables until the export completes with no OOM events.

Example Helm values fragment:

queue:
  env:
    - name: DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB
      value: "30000"
    - name: BULK_EXPORT_MAX_CONCURRENT_RUNS
      value: "3"

Recovering a currently frozen export

  • Stuck partitions auto-clear after ~48 hours.

  • For faster recovery, you can restart the pods with stuck partitions so the export resumes immediately.