LangSmith LLM evaluators scoring 0.0 when root run has no messages

Last updated: June 30, 2026

Symptom

An online LLM evaluator (for example a task_completion evaluator) keeps scoring traced agent runs 0.0 even though the agent produced a correct response. The evaluator reasoning says things like:

  • "the agent provided no output at all (outputs: null)"

  • "The provided conversation is empty."

In the trace UI you can see the response, but it lives inside a child span (e.g. router_chat_stream) or in a custom field like output.response.message, not on the root run.

After partial fixes you may see the opposite failure: the evaluator scores 1.0 with reasoning like "no human requests were made within the provided conversation". That means the human turn is missing from what the evaluator sees too.

Cause

Online evaluators for runs and threads read from the root run's inputs.messages and outputs.messages. They do not walk the run tree to find messages in child spans, and they do not read arbitrary custom fields.

If you use a custom tracing wrapper (for example an endpoint decorated with @trace_endpoint that delegates to a router_chat_stream child span), the final assistant message often lives only in the child's outputs or in a custom response object. The root run's outputs.messages stays null and the evaluator has nothing to score.

Common compounding issues:

  • outputs.messages null at root: the final AI response is only on a child span.

  • inputs.messages missing at root: the root inputs only contain a raw message string, not a messages array of {role, content}. Without that array, {{all_messages}} and human_ai_pairs drop the human turn.

  • Wrong evaluator variable: the prompt references a non-standard path like output.response.message instead of {{all_messages}}.

  • session_id missing on root run: thread-level evaluators use session_id to group runs into a thread. Without it, the conversation can't be assembled.

  • present_inputs is display-only: the present_inputs parameter on @trace_endpoint only changes what the UI shows. It does not mutate ls_run.inputs, so it does not fix the evaluator.

  • ls_run.patch() mid-run: calling patch() to force-push inputs ends the run prematurely and outputs don't get recorded.

  • Decorator cleanup overwriting update_run(): if you call Client().update_run() to set inputs, the decorator's own cleanup PATCH at run end can overwrite them with the stale ls_run.inputs. Keep both in sync.

If some runs in the same project score correctly, it's because those code paths happened to land the assistant message on the root run. The inconsistency is architectural, not random.

Resolution

Fix this in two places: the evaluator config, and your tracing code.

1. Use {{all_messages}} in the evaluator prompt

Change the conversation variable from any custom path (e.g. output.response.message) to {{all_messages}}. That template variable collects messages from both inputs.messages and outputs.messages on the root run into a single conversation list, so the judge sees the full human/AI exchange.

2. Put messages on the root run's inputs and outputs

Set ls_run.inputs to include a messages array before the body runs. Do not call ls_run.patch():

ls_run = get_current_run_tree()
if ls_run is not None:
    ls_run.inputs = {
        "messages": [{"role": "user", "content": message}],
        # other keys are fine to keep
    }

When streaming completes, end the run with the same messages key in outputs. You can keep your existing response schema alongside it:

eval_outputs = {
    "messages": [{"role": "assistant", "content": accumulated_text}],
    "response": final_entry.model_dump(),
}
if ls_run:
    ls_run.end(outputs=eval_outputs)
await trace_session.complete(outputs=eval_outputs)

For interrupted or handoff runs:

await trace_session.complete_for_handoff(
    outputs={
        "messages": [{"role": "assistant", "content": interrupt_entry.message}],
        "interrupt_saved": True,
    },
)

3. Set session_id on the root run for thread-level evaluators

Thread evaluators need session_id (your thread or chat ID) on the root run to group turns into a multi-turn thread:

if not getattr(ls_run, "session_id", None) and chat_id:
    ls_run.session_id = str(chat_id)

4. If the decorator's cleanup PATCH keeps overwriting inputs

Use Client().update_run() to push both inputs and outputs, and keep ls_run.inputs in sync so the decorator's final flush doesn't undo it:

final_inputs = {
    "messages": [{"role": "user", "content": message}],
    "message": message,
    "chat_id": chat_id,
}
eval_outputs = {
    "messages": [{"role": "assistant", "content": accumulated_text}],
    "response": final_entry.model_dump(),
}

if ls_run:
    ls_run.inputs = final_inputs  # keep in sync for decorator's flush
    try:
        from langsmith import Client as LangSmithClient
        LangSmithClient().update_run(
            run_id=ls_run.id,
            inputs=final_inputs,
            outputs=eval_outputs,
        )
    except Exception:
        logger.warning("failed to update run inputs/outputs", exc_info=True)
    ls_run.end(outputs=eval_outputs)

await trace_session.complete(outputs=eval_outputs)

Scope

This applies to any LangSmith user who:

  • Uses online LLM evaluators (run-level or thread-level) on traced agents, and

  • Has a custom agent architecture where the final assistant message lives in a child span or a custom output field instead of the root run's messages, and/or

  • Has an evaluator prompt that references a non-standard variable instead of {{all_messages}}.

Native LangGraph apps that return the full messages list as graph state usually don't hit this, because the state propagates to the root run automatically.

References