Public benchmarks show which model scores higher on a shared test. Only evals built from your own workflow show whether a model change makes your work better, faster, or cheaper.
Agents grow faster than anyone checks them.
Most agents start as a system prompt that describes the job. Then come new features, exceptions, customer preferences, and rules added after something went wrong. As the prompt grows, a team needs to know whether the agent still does the job well, which means defining good work in concrete terms. Aamir Poonawalla brings that habit from Instacart, where the work was cutting cost per delivery without hurting the customer experience.
Public benchmarks are a starting signal.
Benchmarks compare models on a fixed set of tasks. Artificial Analysis and Terminal-Bench, for example, score models and agents on coding, reasoning, and other computer work. When a new model posts a higher score, that is a reason to try it on your own tasks. It does not show the gain will carry over. The real test is whether developers finish sooner, fix fewer errors, or reach the result for less.
Build a benchmark from your own work.
Without a company-specific benchmark, teams keep frontier workflows on the familiar model, because nobody knows whether the agent will keep working on an open-weight one.
Orchestra gathers samples of the tasks each agent sees most often. A customer-record agent's set might hold a contact swap, a pricing question, and an out-of-office response. Evals then turn expectations into checks:
Did the agent pick the right record?
Did it convert a monthly contract amount into the correct annual contract value?
Did the change save?
Did it ask for clarification when the request was unclear?
Some checks run automatically. Others, such as whether an update faithfully reflects its source, need a separate review.
Test what changed, not what the agent says.
Models interpret instructions differently and vary between runs, so evals catch drift whenever instructions, tools, or the model change. For a CRM agent, that means a test environment with a duplicate customer database plus editing tools, so the team can inspect the actual change. An agent reporting that it updated a record does not prove it updated the right one.
With checks in place, each version can be compared on checks passed, run time, and total model cost, retries and corrections included. In one customer-record workflow, an added review step made each GLM run more expensive without any change of model. The checks improved, but new cases and other quality requirements still needed testing before any switch. The full essay includes the run-by-run chart and method note.
Evals compound over time.
Once a cheaper model meets the quality bar on the work that matters, it becomes the better choice. Evals preserve a company's definition of good work through model and infrastructure changes. Orchestra helps teams build those benchmarks and test whether each new model improves quality, speed, or cost.
Start by writing down what good work looks like for one agent. Evaluate your workflows.