Trace retention on self-hosted LangSmith: UI expiry vs. physical deletion

Last updated: October 15, 2026

Overview

Self-hosted LangSmith's retention setting controls when a trace is logically expired, not when its data is physically removed. Once a trace's TTL elapses, it disappears from the UI and from normal trace queries, but the underlying rows can still exist in ClickHouse, and the trace payload can still exist in blob storage, until separate cleanup processes catch up.

This is expected behavior, not a bug. This article explains the three layers involved (UI expiry, ClickHouse physical deletion, blob storage lifecycle) and how to verify the physical cleanup completed.

Why a trace can still be queried after it "expires"

When a trace's retention window ends, it's marked expired based on its TTL tier (short-lived or long-lived) and excluded from normal queries and the UI. That's a logical expiry, sometimes called a soft delete. It happens immediately once the TTL is reached.

The physical removal of the row is a separate, later step, handled by a scheduled cleanup job described below. Until that job processes the relevant data, the row can still be read if you query ClickHouse directly instead of going through the normal trace APIs/UI.

How physical cleanup works

Two independent mechanisms are involved:

  • ClickHouse native TTL deletes whole data parts, not individual rows. A part isn't dropped until every row in it qualifies for deletion, so a part containing a mix of expired and non-expired rows can stick around for a while even though some of its rows are past their TTL.

  • The weekend ClickHouse TTL cleanup cron job (available as of self-hosted LangSmith 0.11, controlled by ENABLE_CLICKHOUSE_TTL_CLEANUP_CRON, enabled by default) runs delete mutations to catch rows native TTL missed. It processes tables one at a time, and within a table it only deletes from a given data part once that part holds at least a configured minimum number of expired rows (100,000 by default). Parts with fewer expired rows than that threshold are skipped until more rows in them expire, which can leave some expired rows queryable for a longer stretch. Rows missing TTL metadata aren't eligible for this cleanup at all.

Neither mechanism touches blob storage. The cleanup cron only deletes from ClickHouse's runs table and related tables. If blob storage is enabled, trace payloads stored there need their own bucket/container lifecycle rules, matching the ttl_s/ (short-lived), ttl_l/ (long-lived), or any custom prefixes you use, to actually be removed. Without those rules, blob payloads for expired traces persist indefinitely.

Backups and retained object versions are independent of all of this. A successful TTL cleanup run or configured blob lifecycle rule doesn't remove copies sitting in backups or object-storage snapshots. Account for those separately if you need a guaranteed full purge.

One related point worth knowing: traces pulled into a dataset are stored as dataset examples with indefinite retention. That only applies to the dataset copy. The original source traces or threads still follow normal TTL and aren't exempted from expiry just because an example was derived from them.

Configuration

Keep the ClickHouse TTL cleanup cron enabled so expired rows actually get purged. The schedule, the minimum-expired-rows-per-part threshold, and the max concurrent mutations are all configurable; see the TTL documentation linked below for the specific environment variables.

If blob storage is enabled, add bucket or container lifecycle rules for the retention prefixes your deployment uses. This is required for traces to be fully removed from blob storage; the TTL cleanup cron won't do it.

Verifying the cleanup job actually ran

To confirm the weekend cleanup completed for a given date window, query ClickHouse's system.mutations table:

SELECT *
FROM system.mutations
WHERE command LIKE '%DELETE%'
  AND command LIKE '%trace_first_received_at%'
  -- add your date range filter here

If every matching row has is_done = 1, the TTL delete mutations for that window completed. Any row with is_done = 0 means a delete mutation is still in progress or stuck, and the corresponding expired data hasn't been physically removed yet. That's worth investigating further before assuming cleanup finished.

References