Tracing project disappears from Tracing tab after an evaluation script upsert

Last updated: August 7, 2026

Symptom

A LangSmith tracing project stops showing up in the workspace's Tracing tab. The project was not deleted: its traces are intact, and the project still opens from its direct URL (by project ID). It now lives under Datasets & Experiments → <dataset name> → <project name>.

The audit log shows update_tracer_session events for the project around the time it disappeared, and no delete_tracer_session or delete_tracer_sessions events.

Cause

An evaluation or experiment script called client.create_project() with the name of an existing live tracing project, a reference_dataset_id, and upsert=True. Because the name matched an existing session and upsert was on, LangSmith updated the existing tracing project in place and attached it to the dataset. That re-parented the project from Tracing into Datasets & Experiments, hiding it from the Tracing tab without deleting it.

The typical trigger is an eval script reusing a production tracing project name (for example, the value of LANGSMITH_PROJECT) as the experiment project name.

Resolution

Platform fix (LangSmith SaaS)

A server-side guard now prevents session upserts from changing a project's type in either direction (tracing project ↔ experiment). Upserting a reference_dataset_id onto an existing tracing project returns a LangSmithConflictError and leaves the existing project unchanged. This applies to both SDK and direct API calls.

Relevant version: LangSmith server 0.17.5. Deployments older than this (self-hosted LangSmith before 0.16) are still exposed to the conversion behavior.

Recover an already-converted project

Projects converted before the guard was released are not restored automatically.

  • Recreate the tracing project using the same name. New traces will route to the new project.

  • Historical traces from the converted project stay accessible by opening the project directly via its UUID-based URL, or under Datasets & Experiments.

Prevent this in evaluation scripts

Use a project name that is dedicated to evaluation and does not match any live tracing project. Do not pass a production tracing project name as project_name when also supplying reference_dataset_id.

Safe pattern:

client.create_project(
    "my-eval-experiment-run-001",
    reference_dataset_id=dataset.id,
    upsert=True,
)

Unsafe pattern (reuses a live tracing project name):

import os
project_name = os.environ["LANGSMITH_PROJECT"]  # e.g. "production-traces"

client.create_project(
    project_name,
    reference_dataset_id=dataset.id,
    upsert=True,
)

Diagnosing with the audit log

If a tracing project disappears and you suspect deletion, check the audit log for update_tracer_session on the project at the time it vanished. An update event (with no matching delete) means the project was re-typed, not deleted. The entry identifies the user or API key that initiated the change.

References