The golden rules

1. One major version at a time — no skipping. Confirmed in self-host-upgrades: 0.10 → 0.11 → 0.12 → 0.13 → 0.14 → 0.15 → 0.16.

2. Land on the latest STABLE patch of each major, never an RC. Target patches: 0.11.29 → 0.12.37 → 0.13.43 → 0.14.x (confirm) → 0.15.17 → 0.16.14. Always verify the real chart+app version currently deployed on both prod and non-prod — don't trust the believed version.

3. Unknown Helm keys are silently ignored on RCs but HARD-FAIL on stable. This is why an RC "works" and then helm upgrade to stable errors on render. Always helm template ... --version <target> to catch config hard-fails before applying — especially important given the values-schema drift accumulated across the 0.10.x line.

4. Backup Postgres + ClickHouse + blob storage before EVERY hop. Migrations are one-way; downgrades are not supported. A fresh backup per hop is the only real safety net. (disaster-recovery)

5. Migration jobs run once per hop → use a change window. From 0.14+ migrations run as a pre-upgrade hook (they gate the rollout), so a failed migration blocks the version bump rather than half-applying.

6. Validate in non-prod first, mirroring prod, then repeat the identical sequence on prod. Validate features in phases: core tracing → Fleet → Insights → Chat.

7. Dependency floors: Postgres ≥ 14, Redis ≥ 6.2 (or Valkey 8), ClickHouse ≥ the version pinned in the target chart. Confirm these are met before the first hop.

Feature flags — keep ALL add-ons OFF until you reach the final version

Rule: carry a values file through every hop that explicitly disables every optional feature, and only turn them on — one at a time — once you are on the final target version and core LangSmith is validated. This is the single most important thing to avoid a failed hop.

Why this matters: several features are enabled by default in recent chart versions. Per the Kubernetes install docs: "Insights (AI-powered trace analysis) and Polly (in-workspace chat) are enabled by default in recent chart versions and require encryption keys at installation time." If you upgrade without explicitly disabling them, the chart will demand encryption keys and dedicated databases mid-upgrade — a hop can hard-fail on a feature you never intended to use yet.

Values to set explicitly (keep these in the values file for the whole upgrade chain)

Legacy config.* flags that must be off if migrating off the old bootstrap model: config.agentBuilder, config.insights, config.polly. These are distinct from the top-level fleet/insights/polly blocks.

Features that are ON by default and are OK to leave on (but review)

• LLM Auth Proxy — on by default since 0.13.37, with a URL allowlist. If Playground/model calls go to hosts not on the allowlist, they get blocked. Review the allowlist before the 0.14 hop. If it runs without a public ingress, it needs SSRF_ALLOW_K8S_INTERNAL on all services making LLM calls plus SSRF_ALLOW_PRIVATE_IPS_PLAYGROUND on playground (0.15).

• ABAC + audit logs — GA and enabled by default since 0.13.37. Fine to leave on; just confirm existing users aren't unexpectedly restricted.

⚠ Encryption-key rule (when you DO enable Insights/Polly/Fleet at the end)

Generate each key once (openssl rand -hex 32) and never change it — insights.encryptionKey, polly.encryptionKey, and the Fleet key must be stable forever. Changing a key permanently corrupts all existing encrypted data with no recovery path. All three features may share one key. Each feature also needs its own dedicated Postgres database (they can share the RDS instance but the chart hard-fails if two features share a database URL).

Order to enable at the end (only after core tracing is green on the final version)

1. Fleet (fleet.enabled, fleetToolServer.enabled, fleetTriggerServer.enabled) → validate

2. Insights → validate

3. Chat / Polly → validate

4. (Optional, license-gated) Engine, Sandboxes

Per-hop breaking changes

Hop 0: 0.10.25 → 0.11.29 (target = latest 0.11 stable)

• Quickwit is removed entirely. 0.10.x ran a Quickwit search cluster (metastore, indexer, searcher, janitor, control-plane StatefulSets, plus an update-runs-index-job). At 0.11 all of those templates — and the whole quickwit: values block — are deleted; runs search consolidates onto ClickHouse. Action: remove every quickwit.* value from your values file, and plan to decommission the Quickwit StatefulSets and their PVCs after the hop succeeds. Verify search/filtering works post-upgrade.

• Large values.yaml rewrite (~430 lines added / ~610 removed). This is the single biggest schema-churn hop — helm template --version 0.11.29 first, and expect renamed/removed keys.

• New feedback data migrations run at this hop (feedback-config-migration + feedback-data-migration Jobs). This is a real data migration — back up Postgres first and budget for a migration window (see downtime section).

• New ingress options added: gateway and istioGateway blocks (Istio Gateway API — HTTPRoute / VirtualService); the old listener/config-map.yaml is gone. No action if you keep your current ingress, but the render will show the new templates.

• e2e post-upgrade test Job introduced, e2eTest.enabled: true by default — this is the job behind the "Run not found" trap noted at the end of this doc.

• New common values you may want to adopt: clusterDomain, namespace, commonInitContainers, commonVolumes, commonVolumeMounts, commonPodAnnotations.

• operator.templates exists from chart 0.11.30 — per-deployment Postgres node-pinning fixes become available once past that patch (relevant later if you enable LangSmith Deployments).

Hop 1: 0.11.29 → 0.12.37 (target = latest 0.12 stable)

• A validate.yaml config guard is introduced in 0.12 — this is where "silently ignored on RC, hard-fail on stable" begins. It fails helm rendering on deprecated/conflicting config. Two rules will bite a deployment coming from 0.10/0.11:

  • config.apiKeySalt becomes required. It previously defaulted to your license key. If you don't explicitly set it (to the same license key), existing API keys are invalidated. Set config.apiKeySalt to your license key before this hop.

  • config.rootDomain is removed → replaced by config.hostname. Your 0.10.25 values still carry rootDomain; leaving it set now hard-fails the render. Migrate to config.hostname.

  • Also enforced: exactly one of ingress / gateway / istioGateway enabled; config.authType must be mixed or oauth; basic-auth and OAuth cannot both be enabled.

• Ingest queue split out — a new optional ingestQueue deployment (enabled: false by default, port 1989) separates ingestion workers. No action required unless you choose to enable it.

• commonPodSecurityContext added.

Hop 2: 0.12.37 → 0.13.43

• Postgres/Redis version trap: 0.13.10 bumped the bundled PostgreSQL and Redis versions, which was breaking and got reverted in 0.13.13 back to Postgres v14.7 / Redis v7. Do not land on 0.13.10–0.13.12. Go straight to 0.13.43. (Your Redis is in-cluster, so this trap directly applies; your RDS Postgres is external, so the PG bundle change is less impactful — but still avoid the window.)

• 0.13.0 first added Agent Builder (later renamed Fleet) support for self-hosted — not relevant unless enabled.

Hop 3: 0.13.43 → 0.14.x

• Breaking: host-backend now picks up commonEnv (previous bug). May produce duplicate environment variables that must be removed from the values file.

• Behavior change — LLM Auth Proxy ON BY DEFAULT with a URL allowlist. If Playground / model calls are used, hosts not on the allowlist get blocked. Review before upgrading.

• ABAC + audit logs GA, enabled by default — verify this doesn't unexpectedly restrict existing users.

• Migrations move to a pre-upgrade hook (safer rollouts). Valkey now a drop-in Redis replacement; GCS Workload Identity supported.

Hop 4: 0.14.x → 0.15.17 — the richest set of gotchas

• Agent Builder → Fleet rename. Hard-fail key renames in the chart:

  • agentBuilderToolServer → fleetToolServer

  • agentBuilderTriggerServer → fleetTriggerServer

  • (Silently ignored on RC, hard-fail on stable — the classic "worked on rc, breaks on 0.15.x" trap.)

• agent-bootstrap script deprecated. Set backend.agentBootstrap.enabled: false — leaving it true causes a duplicate bootstrap that fails. Fleet agents are now standalone Helm-deployed services.

• Workload identity: the Fleet rename may require updating service accounts if workload identity is used.

• New required top-level blocks: fleet:, insights:, polly: must be present. Each needs:

  • its own dedicated Postgres database (can share the RDS instance, but the chart hard-fails if two features share a URL), and

  • a Fernet encryption key (all three can share one key).

  • For the upgrade set all three enabled: false, then flip on one at a time after core is green.

• operator.templates.db must carry node pinning (nodeSelector), or the per-deployment Postgres pod (lg-<id>-0) lands on the wrong node → checkpoint/state connection timeouts while Fleet appears healthy.

• GO_ENDPOINT port trap: :1986 is the Python backend; the Go platform-backend is :8080. Wrong port → 404s on OAuth/OIDC endpoints in 0.15.x. Verify.

• Permission split: projects:update-retention → projects:increase-trace-tier + projects:decrease-trace-tier. Custom roles referencing the old permission need updating.

• Env var rename: USAGE_EXPORT_ADMIN_EMAILS → INSTANCE_ADMIN_EMAILS.

• LLM Auth Proxy without a public ingress: must add SSRF_ALLOW_K8S_INTERNAL to all services making LLM calls and SSRF_ALLOW_PRIVATE_IPS_PLAYGROUND to playground, or SSRF protection blocks private-IP requests.

Hop 5: 0.15.17 → 0.16.14

• agent-bootstrap fully deleted (not just deprecated). If Fleet was ever deployed via the control-plane path, migrate to standalone Fleet first. (migration KB)

• Image consolidation: go-backend, playground, and host-backend are folded into the smith-backend image. Remove any overrides/pins for those images from values.yaml — they no longer exist as separate images.

• New backfillCheck job gates the upgrade until required checks complete. If IAM-based auth is relied on in other services, matching annotations/labels may be needed or the upgrade won't proceed.

• Bulk exports default to zstandard compression when compression is omitted — only matters if export output is parsed downstream.

• New (opt-in, license-gated): SmithDB (public beta — do not self-setup), self-hosted Engine, Sandboxes. None required for a core upgrade.

Backups (run before EVERY hop)

There is no official all-in-one backup script — the Helm chart ships only diagnostic/migration/support scripts. Backups are the operator's responsibility, and the recommended path is native snapshot tooling, because it is consistent and restore-tested:

• Postgres (external RDS): managed snapshot or PITR (aws rds create-db-snapshot ...). Fast.

• ClickHouse (in-cluster, 2 Ti): native BACKUP DATABASE ... TO Disk/S3, or an EBS volume snapshot. Restore time scales with the 2 Ti volume — benchmark on non-prod so you know the real number.

• Blob storage: S3 object versioning + a replicated DR bucket, or aws s3 sync.

Key rules:

• Back up Postgres + ClickHouse + blob to the closest coordinated point in time — restoring Postgres ahead of ClickHouse produces dangling project references and missing traces.

• For a clean snapshot, scale langsmith-queue and langsmith-ingest-queue to zero first, then restore replicas after.

• Verify restorability in non-prod — an untested backup is not a backup.

Migration duration & downtime

There are two separate phases per hop, and only the first is "near-zero downtime":

1. App version roll (near-zero downtime). The stateless pods (smith-backend, frontend, queue workers) do a Kubernetes rolling update. With multiple replicas this is effectively seamless.

2. The database migration (NOT zero downtime). From 0.14+ migrations run as a Helm pre-upgrade hook — the migration Job runs to completion before the new pods roll, so the rollout is gated behind the migration, not overlapped with it safely. During that window:

   • Postgres (RDS): most migrations are DDL that completes in seconds to minutes, but a lock-taking ALTER blocks queries → the still-running old pods error for that window.

   • ClickHouse (2 Ti, in-cluster): most hops are quick DDL, but any hop that rewrites or backfills trace data scales with your 2 Ti volume. For those hops you should pause ingestion and expect reads against affected tables to be degraded/unavailable — that is real downtime proportional to data size, not near-zero. (Note: the 0.10→0.11 hop runs feedback data migrations, and the ClickHouse→SmithDB effort in DMLP-1398 is a heavy data move.)

Bottom line: treat every hop as a booked maintenance window. Schema-light hops will feel near-instant; a data-rewriting hop on 2 Ti will not. We can't give reliable per-hop numbers without benchmarking your environment — so time each hop on a prod-mirroring non-prod first; that dry run IS your prod duration and downtime estimate.

Recovery if a hop fails

• Downgrades are not supported — recover forward from the last good backup, not by rolling back.

Pre-flight checklist

•  Confirmed actual chart + app version on both Prod and non-prod (not the believed version)

•  Postgres ≥14 (RDS), Redis ≥6.2 / Valkey 8 (in-cluster), ClickHouse ≥ target-chart version

•  Fresh RDS + ClickHouse + S3 backups verified restorable (test in non-prod)

•  Change window booked (migrations run once per hop)

•  Non-prod mirrors prod and is upgraded/validated first, hop by hop

•  helm template --version <target> render passes at each hop before helm upgrade

•  Multi-replica confirmed for near-zero-downtime pod rolls

•  Scale numbers (concurrent users, traces/sec) checked against self-host-scale

Known post-upgrade trap

• e2e "Run not found after 15 attempts" at the post-upgrade e2e stage → caused by CLICKHOUSE_ASYNC_INSERT_WAIT_PCT_FLOAT=0 (fire-and-forget inserts; read outruns flush + silent flush failures). Only relevant if you have set that tuning flag. Flip to 1 to fix/surface.

Reference

• Self-hosted changelog: https://docs.langchain.com/langsmith/self-hosted-changelog

• Upgrade steps: https://docs.langchain.com/langsmith/self-host-upgrades

• Latest stable verified 2026-09-01: 0.16.14 (2026-08-28), 0.15.17 (2026-07-27)