One workload.
More ways to improve it.
Start with the model your team trusts. Orchestra turns production work and expert judgment into evaluations, better routes, and specialist models you own.
Start with a workloadSee the integration- 01
Observe
Production traces and expert corrections
- 02
Evaluate
A task-specific definition of good
- 03
Improve
Prompts, models, and specialist training
- 04
Deploy
Reviewed changes and a retained fallback
Evaluation
Your team defines
what good means.
A generic leaderboard cannot tell you whether a model updated the right record or followed your policy. Start with traces and corrections your team chooses to provide, preserve a held-out set, and score the outcomes that matter to your product.
Inspect the published resultsResolve an operations request.
- Correct information
- Required fields match the source request.
- Valid structure
- The output satisfies the downstream schema.
- Intended action
- The selected operation reaches the expected state.
Compare every candidate against the same contract.
Improvement
Use the smallest change
that moves the work forward.
Training is one tool in the process. Often the first useful improvement is a clearer instruction or a better model for the task.
Improve the instructions.
Repair prompts, schemas, parsers, and tool contracts. Make the task precise before asking a different model to do it.
Prompt · schema · tool contractLet models compete.
Compare frontier and open models on the same held-out work. Evaluate quality alongside response time and the cost of the complete route.
Quality · latency · costTrain when the work earns it.
Use accepted traces and expert corrections to train a specialist for the recurring task. Test the resulting model against the same quality bar.
Traces · corrections · specialist weightsRouting and rollout
A better route has
to earn its place.
Keep frontier capability where it changes the outcome. Review a candidate against the baseline, introduce it gradually, and retain the prior route when the evidence calls for it.
- 01
Shadow
Compare candidate behavior alongside the existing route.
- 02
Canary
Introduce a bounded share of traffic after review.
- 03
Ramp
Expand the route as the evidence supports it.
The existing baseline remains part of the decision.
See how the route is behaving.
Operational visibility puts request errors, latency, and usage in view. Evaluation quality is a separate question: a successful request still needs to do the right work.
Ownership
Keep what
your work creates.
Build a body of intelligence your team can continue to improve. Ownership and handoff follow your engagement agreement and the licenses of the underlying models.
- Prompts
- The instructions and task context refined against your work.
- Evaluators
- The examples, rubrics, and checks that define a good outcome.
- Routing rules
- The decisions that put a model to work and keep a fallback available.
- Specialist models
- The weights trained for your task, subject to the underlying model license.
Routed frontier models remain their providers’ models. Your task-specific artifacts and trained specialists are the work you keep.
Integration
Keep the calls
you already make.
Use the existing OpenAI or Anthropic SDK with an Understudy API key and the gateway base URL. Configure your provider or managed-model access in the app.
Orchestra uses the existing Understudy gateway and account system. SDK path conventions differ: OpenAI includes /v1; Anthropic does not.
OpenAI SDK Python
from openai import OpenAI
import os
client = OpenAI(
api_key=os.environ["UNDERSTUDY_API_KEY"],
base_url="https://api.understudylabs.com/v1",
)Anthropic SDK Python
from anthropic import Anthropic
import os
client = Anthropic(
api_key=os.environ["UNDERSTUDY_API_KEY"],
base_url="https://api.understudylabs.com",
)These snippets initialize a client. Use a model enabled for your account when making a request.