Teaching a CRM agent to finish the job
We wanted a lower-cost model to handle everyday CRM work: updating contacts, creating opportunities, and qualifying leads. The test was whether it could leave the right changes in the CRM.
A prompt that worked elsewhere fell short
We tested Qwen 3.6 Plus and Sonnet 4.6 on seven CRM tasks from AutomationBench. The grader checked the actions each agent took and the records it left behind. An attempt passed when it satisfied every requirement.
Qwen started behind Sonnet. We tried adapting prompts that had improved its performance on sales reasoning tasks, but its CRM results got worse. Those prompts focused on choosing the right records. The CRM agent also needed instructions for completing writes and confirming that they had succeeded.
Failed runs showed us what to change
We read the failed attempts and rewrote the prompt around the steps the agent missed. First, we required evidence that each write had gone through, giving the agent a way to check that it had finished the requested work.
Next, we added instructions to use the latest pricing and check account health before creating an opportunity. We kept the model weights, step limit, and token limit fixed while changing the instructions.
The revised prompt brought Qwen ahead of Sonnet on mean task score, at about a quarter of the model cost. The failed actions had shown us which parts of the job we needed to explain.
- Sonnet 4.6
- $1.12
- Qwen 3.6 Plus · revised prompt
- $0.27
Model spend per seven-task AutomationBench run. Qwen’s revised prompt also achieved a 13.1% higher mean task score than Sonnet.
Try this on your workload
Define the records a completed task should leave behind. Inspect failed attempts, then add instructions for the missing actions and their verification. A clear definition of finished work can make a cheaper model more useful.
Methods and full results
Seven AutomationBench CRM tasks. Qwen v5: 10 runs at $0.2749 each. Sonnet 4.6: 3 runs at $1.1204 each.
Verify the records the agent changes
We used seven CRM tasks from Zapier AutomationBench: update a contact’s phone number, add a contact to a campaign, create a note, create a contact for an account, create an opportunity, advance an opportunity’s stage, and qualify a lead. The agent works through the limited_zapier toolset, and the grader checks the records it leaves behind.
We report two scores. Mean score gives partial credit, from 0 to 1, for the checks an attempt satisfies. Strict pass is the share of attempts that satisfy every check. A run is one pass through all seven tasks.
Instructions from failed runs raised the score to 0.630
With its original prompt, Qwen 3.6 Plus scored 0.400. We first tried the prompts we had built for the sales reasoning tasks described below, and both did worse: a hand-written adapter scored 0.369 and the GEPA v3 prompt scored 0.277. Those prompts are about choosing the right records. The CRM tasks needed instructions for finishing writes and confirming them.
We read the failed runs and added instructions for the steps the agent missed. Version 4 required evidence that each write went through, which raised the mean to 0.468. Version 5 added two rules: use the latest pricing, and check account health before creating an opportunity. That brought the mean to 0.630, and cost per run fell from $0.3109 to $0.2749.
These CRM revisions changed only the prompt. The model weights, the ten-step limit and the 4,096-token limit stayed the same.
Qwen 3.6 Plus prompt versions on the CRM tasks
| Prompt | Mean score ± SD | Runs | Cost / run |
|---|---|---|---|
| Original prompt | 0.400 ± 0.067 | 3 | $0.2297 |
| v3 · borrowed from the sales tasks | 0.277 ± 0.056 | 3 | $0.2483 |
| v4 · evidence of completed writes | 0.468 ± 0.077 | 10 | $0.3109 |
| v5 · adds pricing and account-health rules | 0.630 ± 0.087 | 10 | $0.2749 |
Higher scores at 75.5% lower model cost
With the v5 prompt, Qwen’s mean score was 13.1% higher than Sonnet 4.6’s: 0.630 against 0.557. The gap is wider on strict pass. Qwen v5 satisfied every check on 54.3% of attempts, Sonnet on 19.0%, and Qwen with its original prompt on 14.3%.
Model spend for a seven-task run was $0.2749 with Qwen v5 and $1.1204 with Sonnet, 75.5% less. Dividing mean score by that cost gives Qwen v5 4.6 times Sonnet’s score per dollar. Qwen v5 ran ten times (70 attempts); each baseline ran three times (21 attempts).
CRM tasks: all configurations
| Model and prompt | Mean score ± SD | Strict pass | Runs | Cost / run |
|---|---|---|---|---|
| Sonnet 4.6 | 0.557 ± 0.066 | 19.0% | 3 | $1.1204 |
| Qwen 3.6 Plus · original prompt | 0.400 ± 0.067 | 14.3% | 3 | $0.2297 |
| Qwen 3.6 Plus · hand-written adapter | 0.369 ± 0.082 | 14.3% | 3 | $0.2323 |
| Qwen 3.6 Plus · v5 prompt | 0.630 ± 0.087 | 54.3% | 10 | $0.2749 |
A separate prompt improved sales reasoning
We also tested four AutomationBench sales tasks through the api toolset: excluding records that should be left out, choosing by priority, applying rules the request leaves unstated, and checking facts across records. We developed and evaluated a separate prompt for this task set.
Here we used GEPA, an automated prompt optimizer. Qwen’s original prompt scored 0.084 and Sonnet 4.6 scored 0.160. GEPA’s first prompt reached 0.204, and adding completion instructions to it lowered the score to 0.182. Version 3 kept the structure of GEPA’s second prompt and added an explicit rule for when a task is finished. At temperature 0 it scored 0.313 over ten runs and passed 22.5% of attempts outright; Sonnet passed none. A run cost $0.1276, 18% of Sonnet’s $0.7070.
A larger search, with 50 metric calls instead of 20, scored 0.252 ± 0.144, below v3. The best prompt in this experiment combined GEPA’s structure with a clear completion rule.
Sales reasoning tasks
| Model and prompt | Mean score ± SD | Strict pass | Runs | Cost / run |
|---|---|---|---|---|
| Sonnet 4.6 | 0.160 ± 0.009 | 0.0% | 3 | $0.7070 |
| Qwen 3.6 Plus · original prompt | 0.084 ± 0.073 | 0.0% | 3 | $0.0946 |
| Qwen 3.6 Plus · GEPA, default temperature | 0.157 ± 0.124 | 8.3% | 3 | $0.1015 |
| Qwen 3.6 Plus · GEPA v3, temperature 0 | 0.313 ± 0.110 | 22.5% | 10 | $0.1276 |
Experiment setup
We ran Claude Sonnet 4.6 through Anthropic and Qwen 3.6 Plus through Fireworks. The main comparisons used max_steps=10 and max_tokens=4096. GEPA ran from the standalone gepa package, with Claude Opus 4.7 as the reflection model, max_metric_calls=20 and reflection_minibatch_size=2. We wrote the CRM v5 prompt by hand.
Development and evaluation used the same tasks: seven CRM tasks for v5 and four sales tasks for GEPA v3. Baselines ran three times and final prompts ten times; the tables report variation across runs.
We also tried giving the agent more steps on the sales tasks, using an earlier GEPA prompt. Raising max_steps from 10 to 20 moved the mean from 0.157 ± 0.124 to 0.169 ± 0.144 over three runs each, and raised cost from $0.1015 to $0.1441 per run. GEPA v3 scored 0.313 with ten steps.