Helm upgrade fails on a StatefulSet with "Forbidden: updates to statefulset spec"

Last updated: October 8, 2026

Symptom

A helm upgrade on a self-hosted LangSmith deployment fails partway through with an error like:

error updating the resource "langsmith-clickhouse":
 cannot patch "langsmith-clickhouse" with kind StatefulSet: StatefulSet.apps "langsmith-clickhouse" is invalid: spec: Forbidden:
 updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit',
 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
Error: UPGRADE FAILED: cannot patch "<statefulset-name>" ...

This looks like it's specific to whichever component is named in the error (ClickHouse, Postgres, Redis), but it's a Helm/Kubernetes deployment-update failure, not an application or query error. The rest of the chart upgrade (other Deployments/StatefulSets, migration jobs) typically completes, and only the affected StatefulSet's patch fails. The existing pod keeps running on its previous configuration, so there's no outage or data loss from this error by itself.

Cause

Kubernetes only allows in-place updates to a small set of StatefulSet spec fields: replicas, ordinals, template, updateStrategy, revisionHistoryLimit, persistentVolumeClaimRetentionPolicy, and minReadySeconds. Fields like volumeClaimTemplates (which controls PVC storage size and storage class) and selector are immutable once the StatefulSet is created.

In the LangSmith Helm chart, the StatefulSets for ClickHouse, Postgres, Redis, and the bundled taskdb/agent-features Postgres and Redis all render their PVC size and storage class from values (for example clickhouse.statefulSet.persistence.size and storageClassName) into a volumeClaimTemplates block. If the values used for an upgrade specify a different disk size or storage class than what the StatefulSet was originally created with, Helm's patch to the existing StatefulSet is rejected by the Kubernetes API with this "Forbidden" error, and the upgrade fails at that resource.

This is a generic Kubernetes/Helm constraint, not a bug in a specific LangSmith component. Any customer who changes a StatefulSet's storage size, storage class, or other immutable field in their Helm values before re-running helm upgrade will hit this, regardless of chart version.

Resolution

The StatefulSet needs to be deleted (orphaning its pod and PVC, not cascading to them) and recreated by Helm with the new spec. This preserves existing data.

  1. Delete the StatefulSet with --cascade=orphan: 

    kubectl delete statefulset <statefulset-name> --cascade=orphan -n <namespace>

    This keeps the existing pod and PVC in place. Data on the existing volume isn't touched and nothing is deleted.

  2. Re-run the same helm upgrade command. Helm recreates the StatefulSet with the updated spec (for example, the new storage size), and the existing pod/PVC are reused.

Don't restart or delete the pod to try to fix this: it doesn't resolve the immutable-field conflict. Don't delete the PVC. Don't force the upgrade with a blanket --force, and don't assume a Helm rollback is safe once migration jobs have already run as part of the upgrade.

Before you re-run the upgrade, check whether the values file you're using actually intends the storage size/storage class change. If the change was unintentional (values drift between environments or between who last touched the chart), fix the values first so you don't apply an unwanted resize.

References