Confirming whether custom code evaluators are actually running

Last updated: November 11, 2026

Symptom

A code evaluator rule is configured on a workspace, but there's no direct way from the rule configuration itself to tell whether it's actually being invoked. The Evaluators configuration screen shows what's set up, not what's running.

Cause

Configuring a code evaluator only defines the rule. Actual execution happens at run time, separately from the configuration surface, so you need to look at execution-time signals (UI run history, logs, or last-run timestamps) rather than the rule definition alone to confirm usage.

Resolution

Three ways to check actual usage, in order of effort:

1. UI

Open Evaluators in the sidebar and filter by Type: Code. To see run history for a specific evaluator, open the Evaluators tab on the relevant tracing project.

2. Execution logs

Filter your collected logs for the code-execution service for the message Starting code execution over the period you care about.

  • Each matching log entry is one execution request. Sum the num_executions field on each entry to account for requests that batch multiple function executions together.

  • Where present, group by tenant_id, session_id, and rule_id to attribute activity to specific workspaces, projects, and rules. These fields are supplied by the caller and aren't guaranteed to be present on every entry.

  • These counts reflect execution attempts, not confirmed successful evaluations or distinct users. Available fields can vary by deployed version, and results depend on your log retention and whether logs are being collected across all replicas of the service.

3. Postgres

Query for every code evaluator rule across workspaces and when it last ran:

SELECT t.display_name AS workspace, r.id AS rule_id, r.display_name AS rule, r.is_enabled,
       CASE WHEN r.dataset_id IS NOT NULL THEN 'dataset/experiment' ELSE 'tracing project' END AS target,
       ce.language, r.created_at,
       (SELECT max(b.application_time) FROM run_rules_applications_batches b WHERE b.rule_id = r.id) AS last_run
FROM run_rules r
JOIN tenants t ON t.id = r.tenant_id
LEFT JOIN evaluators e ON e.id = r.evaluator_id
LEFT JOIN code_evaluators ce ON ce.evaluator_id = e.id
WHERE e.type = 'code'
   OR (r.code_evaluators IS NOT NULL AND r.code_evaluators::text NOT IN ('null','[]'))
ORDER BY last_run DESC NULLS LAST;

This returns each code evaluator rule's enabled state, whether it targets a dataset/experiment or a tracing project, its language, and the most recent run timestamp.

None of these approaches cover evaluators run locally through the SDK or an IDE. Those execute outside the platform's code-execution service entirely and won't show up in its logs.

References