Seven tasks that change the CRM.
The action slice covers updating a contact’s phone, adding a contact to a campaign, creating a note, creating a contact for an account, creating an opportunity, advancing its stage, and qualifying a lead. The limited_zapier tool surface gives the agent the tools needed to complete each task.
A generic adapter tuned on reasoning-heavy API tasks did not transfer: its 0.369 mean score was below the raw open model’s 0.400. The targeted v5 adapter added proof-artifact requirements, latest-pricing rules and account-health checks after inspecting failed trajectories.
This result is prompt and adapter optimization. The publication does not report weight training for the v5 CRM result.
Write-heavy CRM slice
| Route | Mean score ± SD | Strict pass | Replicates | Cost / full slice |
|---|---|---|---|---|
| Sonnet 4.6 · frontier ceiling | 0.557 ± 0.066 | 19.0% | 3 | $1.1204 |
| Qwen 3.6 Plus · open baseline | 0.400 ± 0.067 | 14.3% | 3 | $0.2297 |
| Qwen 3.6 Plus · + hand adapter | 0.369 ± 0.082 | 14.3% | 3 | $0.2323 |
| Qwen 3.6 Plus · + targeted v5 adapter | 0.630 ± 0.087 | 54.3% | 10 | $0.2749 |
The reasoning slice is a separate experiment.
Four reasoning-heavy sales API tasks test negative selection, priority selection, implicit rules and cross-reference validation. On this surface, GEPA v3 at temperature zero scored 0.313 ± 0.110 across ten replicates, versus Sonnet’s 0.160 ± 0.009 across three.
The two slices use different task and tool surfaces. Their absolute scores and costs should not be pooled. The v3 adapter exceeded the reference mean with substantial remaining rollout variance.
Reasoning-heavy API slice
| Route | Mean score ± SD | Strict pass | Replicates | Cost / full slice |
|---|---|---|---|---|
| Sonnet 4.6 · frontier ceiling | 0.160 ± 0.009 | 0.0% | 3 | $0.7070 |
| Qwen 3.6 Plus · open baseline | 0.084 ± 0.073 | 0.0% | 3 | $0.0946 |
| Qwen 3.6 Plus · + GEPA (default temp) | 0.157 ± 0.124 | 8.3% | 3 | $0.1015 |
| Qwen 3.6 Plus · + GEPA v3 @ temp=0 | 0.313 ± 0.110 | 22.5% | 10 | $0.1276 |
Method and cost basis.
Benchmark: the sales domain of Zapier AutomationBench. Models: Claude Sonnet 4.6 through Anthropic and Qwen 3.6 Plus through Fireworks. All runs use max_steps=10 and max_tokens=4096.
GEPA used the standalone gepa package with Claude Opus 4.7 as the reflection model, max_metric_calls=20 and reflection_minibatch_size=2. The reasoning-heavy v3 run was replicated to n=10. CRM v5 was a manual failure-mode hill climb, also replicated to n=10.
CRM v5 cost $0.2749 per full slice versus $1.1204 for Sonnet: 24.5% of the reported reference cost. These are evaluation-slice costs. They are not a bill for end-to-end optimization, training or operating the product.
The publication identifies the selected benchmark tasks and run settings. An independent, untouched confirmation set, exact runtime/package pins, confidence intervals and end-to-end optimization cost are not reported in this publication.
Limitations and source record.
- These are narrow benchmark slices, not a claim of general model superiority.
- The optimized and reference rows have different replicate counts; the observed means retain visible variance.
- The same task surfaces informed adapter development. Treat the results as optimization evidence, not proof of unseen-task generalization.
- A 0.630 mean is partial credit. The optimized CRM route’s strict-pass rate is 54.3%.
Source: the original Understudy publication and its checked-in benchmark snapshot. Adapted for Orchestra on 5 September 2026. This page preserves historical evidence; it is not a new run, live telemetry or a forecast.
Read the original publication ↗Benchmark references: Zapier AutomationBench · Prime Intellect introduction