How to copy labels when adding a thread to a dataset and structure them for offline evaluation

Symptom

When you add a thread (for example from an Annotation Queue) to a Dataset, the resulting example contains only the conversation (inputs.messages) and a reference back to the source thread. Feedback applied to the thread during review (tags, scores, category labels, comments) is not copied onto the example, and no reference output is populated. If you open the example in the UI, you won't see the review labels, and if the source trace later expires, that feedback is gone for good.

Cause

Adding a thread to a Dataset is not an archive of its annotations. The import path that creates the example captures the conversation and a provenance link back to the thread - essentially, input.messages - but it does not copy thread feedback or populate the example's outputs. This is specific to thread imports: importing from a run does copy feedback summaries into example metadata, so don't assume the two behave the same way.

Resolution

Copy thread feedback onto the example explicitly, before the source trace or thread data expires. Queue membership alone doesn't extend retention of the underlying trace, only automation does, so don't treat an annotation queue as permanent storage.

• Read the thread's feedback (keys, scores/values, comments) by thread ID via the SDK or API.

• Write it onto the dataset example with update_example, merging into existing metadata rather than overwriting it:

  • Store rubric keys, scores/category labels, comments, and source identifiers in the example's metadata.

  • If a label is the ground-truth/grading target for an evaluator, put it in the example's reference outputs instead of metadata.

• You can do this through the UI by editing the example directly, or through the SDK with update_example.

Once copied, the labels persist in the dataset past the source trace's retention window. They just aren't copied for you automatically, so this step has to happen while the source data is still available.

Structuring offline evaluation on labeled threads

Get single-turn (trace-level) evaluations working before layering on thread-level (multi-turn) evaluations.

• Reshape a thread into a test case: since inputs.messages is what gets copied, move the last message out of inputs and into outputs to create an expected result. The function under test then generates only the final turn, and evaluators score it against that expected result.

• Use the copied reviewer labels as evaluator baselines or test criteria, for example whether a new run resolved a thread the reviewer had marked resolved, or whether it called the correct tool when a reviewer had flagged an incorrect-tool label.

• Split datasets by purpose: keep a regression set of threads that must keep passing separate from a set of hard or failing threads you're actively improving, either as a dataset split or a separate dataset.

• Distinguish two evaluation workflows:

  • Judge calibration: use the full transcript as input and the human label as the reference output, then compare the judge's prediction to that label.

  • Application regression testing: use the conversation history up to a turn as input, with a curated expected reply or acceptance criteria as reference. Don't feed the answer being tested back into the input.

• Keep historical human ratings separate from scores assigned to newly generated responses. Use dataset splits and versioning for repeatable comparisons.

• Once single-turn evaluation works on labeled threads, layer on multi-turn evaluation.

References

• Manage datasets in the application

• Manage datasets programmatically

• Observability concepts

• Code evaluator SDK

• Evaluation concepts

• Annotation queues

• Online evaluations: multi-turn