LangSmith bulk export OOM: tuning buffer size and concurrency

Last updated: July 23, 2026

Status: internal draft, needs human review before publishing. Relevant version: self-hosted LangSmith Helm chart 0.15.x.

Symptom

On a self-hosted LangSmith deployment, triggering a bulk export OOM-kills the langsmith-queue pod during Parquet serialization/upload. The kill lands a few seconds after a Writing Parquet file to S3 log line, inside _write_parquet_to_s3 in smith-backend/app/models/bulk_exports/export.py (~line 938).

After the pod dies, the partition runs it was processing stay in Running. The export log then repeats a line like {Completed: 25, Created: 250, Running: 5} every minute. No new partitions are picked up because the global concurrent-run slots are held by the stuck entries, and the export appears frozen.

Why it happens

Each export partition fetches a full page of DATA_EXPORT_RUN_LIMIT runs (default 500) from the data store and materializes all of them into memory, including full inputs, outputs, and extra fields, before the payload-size guard is evaluated. That materialization is a large temporary raw-memory spike.

The batch sizer computes bytes_per_run from the compressed Parquet file written by the previous batch, not from actual in-memory cost. Compression ratios between raw runs and serialized Parquet are high (observed around 17x), so the sizer can confidently target a 500-run page as "safe" while the true in-memory cost is many times the compressed size. In one reproduction, a batch that produced a ~67 MB Parquet file occupied ~1.15 GB in memory, and the following 500-run page reached ~1.87 GB.

The OOM fires during _write_parquet_to_s3, after the run page has already been materialized. The primary driver is the number of runs materialized per batch (DATA_EXPORT_RUN_LIMIT), not the payload-size ceiling. Lowering the payload ceiling alone does not prevent the initial materialization spike.

Concurrency multiplies the pressure. Up to BULK_EXPORT_MAX_CONCURRENT_RUNS partition runs execute at once (default 5). Each concurrent run carries its own in-memory batch, and they all live inside the same queue process on the same pod, so peak memory is the sum of every active batch plus its serialization buffers. This is a global, per-export limit enforced against a shared Postgres count and serialized by a per-export Redis lock (see smith-backend/app/models/bulk_exports/jobs.py, cron_schedule_bulk_exports lines 1199-1243). It is not a per-pod limit. Adding langsmith-queue replicas does not raise it, and scaling pods is not a concurrency lever for this variable.

Fix runbook

All three variables go on the langsmith-queue deployment only, under queue.deployment.extraEnv in the Helm values file (injected via helm/charts/langsmith/templates/queue/deployment.yaml line 13). Defaults live in the bulk-export block of lc_config/lc_config/settings.py. Apply the steps in order; Step 1 is the lever that actually stops the OOM.

Step 1 — Lower DATA_EXPORT_RUN_LIMIT (primary fix)

This addresses the OOM by shrinking the per-batch memory spike: fewer runs are materialized before the size guard fires.

  • Default: 500.

  • Around 50 cleared batches that consistently OOM'd in the reproduced case; 10 gives maximum safety at the cost of throughput.

  • Tune upward while watching pod memory to find the right value for your trace sizes.

Step 2 — Lower BULK_EXPORT_MAX_CONCURRENT_RUNS (supporting fix)

This addresses the OOM by reducing how many batches are active in memory simultaneously. Fewer concurrent batches serializing at once lowers peak memory.

  • Default: 5.

  • Global, per-export limit. Not per-pod. Scaling langsmith-queue replicas will not change it.

Related variables in the same block, useful to know but not the primary knobs for this fix:

  • BULK_EXPORT_MAX_CONCURRENT_RUNS_WS_OVERRIDES: per-workspace override map.

  • BULK_EXPORT_MAX_CONCURRENT_PARTITION_RUNS: separate global workspace cap.

  • MAX_CONCURRENT_BULK_EXPORTS_PER_WS: cap on simultaneous export jobs per workspace.

Step 3 — Lower DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB (secondary safeguard)

This adds a secondary ceiling on serialized payload size, but does not prevent the initial batch materialization spike. In the investigated case, lowering this alone did not stop the OOMs, because the run page is materialized before this guard applies.

  • Default: 100000 (~100 MB).

  • Recommended: 30000 (~30 MB).

Verified-stable configuration

Confirmed stable over a week in a self-hosted production deployment at 8 Gi pod memory. Conservative and will slow export throughput; tune upward while monitoring pod memory.

queue:
  deployment:
    extraEnv:
      - name: DATA_EXPORT_RUN_LIMIT
        value: "10"
      - name: BULK_EXPORT_MAX_CONCURRENT_RUNS
        value: "1"
      - name: DATA_EXPORT_MAX_BATCH_PAYLOAD_SIZE_KB
        value: "30000"

Known residual behavior

When the pod is OOM-killed mid-partition, the OS delivers SIGKILL. The process cannot perform cleanup or status transitions, so its runs stay in Running. That is why the export log freezes at a fixed Running: N and no new partitions are picked up: the global concurrency slots are held by runs that will never self-resolve via the normal completion path.

Stuck runs auto-clear at a timeout on the order of days. Preventing the OOM (Steps 1-3) is the real fix. Once the OOM stops, runs complete normally and no manual unsticking is needed.

References

  • Monitor and troubleshoot bulk exports

  • Manage bulk export destinations

  • smith-backend/app/models/bulk_exports/export.py ~line 938: full-page fetch, _prepare_run materialization, _write_parquet_to_s3.

  • smith-backend/app/models/bulk_exports/jobs.py lines 1199-1243: concurrency orchestration, Redis lock, per-export Postgres count.

  • lc_config/lc_config/settings.py: bulk-export settings block and env var defaults.

  • helm/charts/langsmith/templates/queue/deployment.yaml line 13: queue.deployment.extraEnv injection point.