Cutting response times for everyday operations
A small model could handle scheduling and incident tasks, but spent too long generating its answers. We investigated what it was writing and found a way to make it much faster.
A short answer was taking too long
An operations agent may need to log an incident in Jira and update its Confluence record. The job calls for the right actions and a short, structured reply.
We tested Qwen3-8B on 30 operations workflows. It handled most requirements, but spent much of its generation on extended thinking and extra text. Producing a short answer involved far more writing than the application needed.
We supplied the opening of the JSON reply so the model could continue directly in the required format. Token use fell 18.8×. The original and fine-tuned models achieved the same task score with this change, so we could get this improvement by controlling the output alone.
Free-form response
The model decides how to open and format its answer.
Here is the classification:
```json
{
"intent": "refund_request",
"priority": "high"
}
```
The customer wants a refund.JSON prefill
The application writes the start of the JSON. The model fills in the rest.
{ "intent": "refund_request", "priority": "high" }
Underlined: written by the applicationBlue: written by the model
Training helped it complete more tasks
Disabling extended thinking reduced output further, but exposed missed steps. We used training examples to improve task completion at about the same token count. Training repaired most errors in the Jira and Confluence incident workflow; vendor onboarding still needed work.
We then served the fine-tuned model on Fireworks and compared it with Sonnet through Anthropic’s API. The smaller model responded about five times faster. The remaining failed tasks gave us a concrete list of work to address before expanding its role.
The small model passed every task check on 92.2% of attempts; Sonnet passed 100%. Both ran 30 AutomationBench workflows, with 90 attempts per model.
Try this on your workload
Check what your model spends time generating. Tightening its response format can deliver a large improvement on its own. Use the remaining failed actions to decide where training will help.
Methods and full results
Check completed actions across 30 operations workflows
We used 30 tasks from AutomationBench’s operations domain: scheduling, safety incidents, legal document archiving, and record updates across tools such as Jira, Confluence, Pipefy and Notion. Each task needs the right sequence of tool calls and a structured JSON response. The scorer gives partial credit for each requirement an attempt meets, and a strict pass means it met all of them.
We trained Qwen3-8B on examples generated by Qwen3-30B, then compared their task scores. The first comparison ran each task three times. Scheduling, safety, and facilities work reached full scores with the 8B model; document operations and cross-system work gave us specific errors to investigate.
Task categories, first comparison
| Category | Includes | Attempts | 8B, fine-tuned + prefill | 30B reference |
|---|---|---|---|---|
| Scheduling and coordination | kickoffs, room conflicts, trainings, maintenance windows | 12 | 1.0000 | 1.0000 |
| Compliance and safety | hazmat, safety incidents, policy notices, sensor alerts | 18 | 1.0000 | 1.0000 |
| Facilities and inventory | calibration, mailroom, fleet, perishable inventory | 21 | 1.0000 | 1.0000 |
| Legal and document operations | leases, DocuSign, NDAs, archive workflows | 12 | 0.8750 | 1.0000 |
| Cross-system workflow execution | Jira, Confluence, Pipefy, Notion, Mailchimp, Slack, Sheets | 27 | 0.9444 | 0.9444 |
JSON prefill cut token use 18.8×
We used the same 30 tasks to evaluate and select methods, running each setup 25 times per task for 750 attempts. With default generation on Tinker, Qwen3-8B scored 0.9560 but used 1,390,498 tokens and took a median 32.6 seconds per response. Extended thinking and extra text accounted for much of the work required to produce a short JSON answer.
JSON prefill supplies the opening of the assistant’s JSON reply. The model continues from that prefix in the format the application expects. Prefill reduced token use to 74,008 and median latency to 6.8 seconds. Mean score reached 0.9667, with 93.33% of attempts passing every check.
The original and fine-tuned models both scored 0.9667 with prefill and passed 93.33% of attempts. Testing the formatting change separately showed that output control delivered this improvement before further training.
Fine-tuning repaired errors after thinking was disabled
Qwen3’s /no_think instruction disables extended thinking. It reduced token use to 38,113, 36.5× fewer than default generation, with a mean score of 0.9598 and 91.20% strict pass. The remaining failed actions identified where the faster configuration needed training.
Supervised fine-tuning (SFT) teaches the model from completed examples. Combined with /no_think, it raised mean score to 0.9733 and strict pass to 94.67%, an increase of 3.47 percentage points at about the same token count. We selected this configuration for the serving test.
Qwen3-8B configurations
| Qwen3-8B setup | Mean score ± SD | Strict pass | Tokens | Median / p95 latency |
|---|---|---|---|---|
| raw no prefill | 0.9560 ± 0.1847 | 94.00% | 1,390,498 | 32.60 / 63.26 s |
| + JSON prefill | 0.9667 ± 0.1248 | 93.33% | 74,008 | 6.76 / 23.48 s |
| SFT + JSON prefill | 0.9667 ± 0.1248 | 93.33% | 74,630 | 6.63 / 26.96 s |
| + /no_think | 0.9598 ± 0.1314 | 91.20% | 38,113 | 5.34 / 21.39 s |
| SFT + /no_think | 0.9733 ± 0.1124 | 94.67% | 39,089 | 5.52 / 23.80 s |
Use failure clusters to choose training examples
Most remaining failures came from three workflows. Fine-tuning mostly fixed the Jira and Confluence incident task and partly fixed the Drive and Notion lease archive. Pipefy vendor onboarding still failed repeatedly. The next round of training data should target these three.
Failures by workflow
| Workflow | What went wrong | Next step |
|---|---|---|
| Jira / Confluence incident | Skipped a required action in one of the two systems. Mostly fixed by SFT. | Add variants of the incident workflow. |
| Drive / Notion lease archive | Finished only part of the document and archive steps. Partly fixed by SFT. | Add complete document-handling examples. |
| Pipefy vendor onboarding | Failed repeatedly, before and after SFT. | Add targeted examples or train against verifier feedback. |
369 ms median responses on Fireworks
We deployed the fine-tuned /no_think model on Fireworks and ran each of the 30 tasks three times. Sonnet 4.6 ran the same 90 attempts through Anthropic’s API. Median response time was 369 ms for Qwen3-8B and 1,935 ms for Sonnet, a 5.2× speedup. Qwen’s p95 was 536 ms.
Qwen3-8B scored 0.9630 and passed every check on 92.22% of attempts. Sonnet scored 1.0000 with 100% strict pass. The comparison below pairs response time with task quality for each serving setup.
Serving comparison
| Model and host | Mean score | Strict pass | Median latency | Token cost, 90 attempts |
|---|---|---|---|---|
| Sonnet 4.6 · Anthropic API | 1.0000 | 100% | 1,935 ms | $0.039969 |
| Qwen3-8B, SFT + /no_think · Fireworks | 0.9630 | 92.22% | 369 ms | $0.006617 |
Measure the cost of the complete job
The Fireworks run used 33,111 tokens, with a recorded model-token cost of $0.006617, compared with Sonnet’s $0.039969: about 6× lower for the 90-attempt job. Including GPU startup time, the short-lived Fireworks deployment cost about $1.56.
The original cost model used $18 per million blended tokens for Sonnet and a $0.20 serving-cost basis for Qwen3-8B, a 90× difference in unit cost. Each model generated a different number of tokens, producing the measured 6× difference in job token cost. For a recurring workload, budget from its measured token use and include the time the deployment stays running.