Routing a decision model evaluator through a custom OpenAI Decisions gateway
Last updated: October 28, 2026
Overview
A decision model evaluator can route its model calls through your own gateway instead of calling OpenAI directly. This works by configuring the evaluator with the OpenAI Decisions Compatible Endpoint provider (listed under "Other" providers), as long as your gateway implements the OpenAI Decisions API request/response format. If your gateway exposes a different format, you need an adapter in front of it.
Configuration
Add the gateway's API key as a workspace secret under Settings > Integrations > Provider secrets (+ Secret). Select OpenAI to store it under the default name
OPENAI_API_KEY, or Custom to use your own secret name.In the decision model evaluator, go to Prompt & Model > Model Configuration and set Provider to OpenAI Decisions Compatible Endpoint.
Enter Model (the model ID your gateway expects), API Key Name (the secret name from step 1), and Base URL (the root of your gateway's API). LangSmith appends
/v1/decisionsautomatically, so don't include it in the Base URL.Click Save as preset, name the configuration, and select it as the evaluator's model.
Map trace/run inputs and outputs into State, add evaluation questions, and save the evaluator.
When the evaluator runs, LangSmith calls <Base URL>/v1/decisions on your gateway and sends the stored secret as a Bearer token.
Related but distinct options
This compatible-endpoint path is separate from calling OpenAI Decisions directly through the LangSmith LLM Gateway's own /openai/v1/decisions route. That route is for routing your application's Decisions calls through LangSmith's gateway, not for configuring an evaluator against an external gateway.
A TypeSafe-compatible endpoint (for servers implementing the TypeSafe System One API, appending /v1/systemone) is a separate option with its own request/response format and defaults. Don't confuse it with the OpenAI-Decisions-compatible endpoint described above.