Detecting experiment completion and fetching final scores in LangSmith

Symptom

You want a reliable signal that a dataset experiment has finished, so downstream code can pull final scores or fire alerts. Reading raw run state directly from storage looks unreliable: runs the LangSmith UI shows as completed still appear with status = 'pending' and a NULL end time in a direct query, and there is no single "experiment complete" marker at the session level.

Cause

Run records are versioned: every update to a run is written as a new version rather than an in-place edit, and the previous version is not removed until the storage engine merges them in the background. A raw query that filters on status = 'pending' and a null end time matches the stale pre-update version, not the current state. The LangSmith UI reads the deduplicated, post-merge view, so it shows the run as done while a raw query on the same run does not.

There is also no session-level "experiment complete" row to key off of. Completion has to be inferred from the runs, or observed from the SDK.

Resolution

Use the SDK. evaluate() (Python) and evaluate (TypeScript) return only after every example and every evaluator has finished, so the return value itself is the completion signal. No polling, no direct storage access.

To fetch the aggregate scores after completion, call read_project / readProject with include_stats=True. The flag is required: feedback_stats is empty without it.

# Python
resp = client.read_project(
    project_name=results.experiment_name,
    include_stats=True,
)
print(resp.feedback_stats)

// TypeScript
const resp = await client.readProject({
    projectName: results.experimentName,
    includeStats: true,
});
console.log(resp.feedback_stats);

The payload is keyed by evaluator:

"feedback_stats": {
  "<evaluator_key>": {
    "n": 9,
    "avg": 0.667,
    "values": { "CORRECT": 6, "INCORRECT": 3 }
  }
}

Prefer the SDK and API over querying LangSmith's underlying storage directly. The API is the stable interface; the internal schema can change between releases and dashboards built on it may break on upgrade.

Data model notes

Experiments and tracing projects share the same underlying object in LangSmith, called a session. A run's session_id is the experiment (or project) ID. A dataset experiment produces one root run per example, multiplied by num_repetitions; each root run has is_root = true and a null parent_run_id. Per-run evaluator scores live on the run's feedback_stats; the aggregate across the whole experiment comes from read_project(..., include_stats=True).

References

• Fetch performance metrics for an experiment