Finding purchase intent in 40,000 comments
A film can attract thousands of enthusiastic comments. We tested an open model’s ability to find people planning to see it in theaters, with full-table model cost of $2.82.
Separate excitement from plans to go
The job was to turn 39,962 YouTube comments from a Snowflake table into labels an analyst could use. We asked each model to identify sentiment, intent, and whether someone explicitly planned to see the film in a theater. That last distinction matters when building an audience of prospective ticket buyers.
We ran the same comments through an Orchestra post-trained 30B open model, Sonnet, and Opus. Each flagged about one percent of comments for theater intent, but they chose different rows. Comparing those positives and disagreements gives reviewers specific examples to examine and a way to refine the attendance rule.
“This looks amazing.”
Positive reaction. No stated plan to see it in theaters.
“Taking my kids opening weekend.”
A stated plan to attend a screening.
Short answers still need room to finish
In a separate prompt experiment, we tightened the attendance rule and removed explanations from the answer. Label scores improved. But our most restrictive version cut some responses off before the model could finish its JSON, leaving rows the application could not use.
We kept the structured prompt and gave the model more room to finish. The selected version improved on the original prompt and returned valid output for every comment in that test.
The open model returned 39,949 valid JSON outputs, with 13 rows to retry. All three models processed the same table.
Try this on your workload
A useful labeling job needs a precise business question and a review loop. Define the labels, test the difficult examples, and give the model enough room to return a complete answer.
Methods and full results
Turn comments into sentiment and attendance labels
We ran three models over 39,962 non-empty YouTube comments stored in a Snowflake table. We asked each model to assign sentiment and intent labels, plus a yes-or-no flag for explicit intent to see the film in a theater. The candidate was a 30B open model post-trained by Orchestra, with Sonnet and Opus as comparisons.
The theater flag is narrower than sentiment. For example, “this looks amazing” is positive but says nothing about buying a ticket, while “taking my kids opening weekend” does. Keeping the two separate lets an analyst measure how people reacted and, from the same table, find the ones planning to go.
Full-table processing for $2.82
The open model processed the table for $2.82. Sonnet cost 4.4 times as much and Opus 49.6 times as much. All three attempted every comment. The open model returned 39,949 valid outputs, with 13 rows to retry. Opus had one parse failure and Sonnet had none.
Full-table results by model
| Model | Model cost | Parse failures | Theater positives | Positive rate |
|---|---|---|---|---|
| Post-trained 30B open model | $2.82 | 13 | 368 | 0.92% |
| Sonnet | $12.48 | 0 | 360 | 0.90% |
| Opus | $139.63 | 1 | 405 | 1.01% |
Use disagreements to focus label review
Three-way agreement is the share of comments where all three models gave matching labels. On the 39,948 comments with valid output from all three, agreement was 69.55% for sentiment, 69.17% for intent, and 99.30% for the theater flag.
Theater positives accounted for about 1% of comments: 368 for the open model, 360 for Sonnet, and 405 for Opus. Most theater flags therefore agree on “no”. Reviewing the positive and disagreement rows shows which comments each model would include in, or miss from, a theater-intent audience.
The publication’s model-derived reference contains 319 strict-consensus positives. The open model found 249 of them and flagged 119 comments outside the reference. Precision was 67.66%, the share of its flags inside the reference; recall was 78.06%, the share of reference positives it found.
Agreement between pairs of models
| Models | Valid rows | Sentiment | Intent | Theater flag |
|---|---|---|---|---|
| Sonnet vs Opus | 39,961 | 85.92% | 81.71% | 99.68% |
| Sonnet vs open model | 39,949 | 72.88% | 76.69% | 99.54% |
| Opus vs open model | 39,948 | 77.62% | 77.76% | 99.39% |
Give structured responses enough room to finish
We tested five prompt variants for the theater flag. Removing the rationale and tightening the attendance rule both raised F1, which combines precision and recall. Those variants had 19 and 23 parse failures. Strict prefill with an 80-token cap reached 0.7912 F1, with 20 outputs cut off before they finished.
We selected strict prefill with a 180-token cap. It reached 0.7895 F1, up from the prompt-test baseline’s 0.7673, with zero parse failures. The larger token budget let the model finish its structured response while retaining most of the label improvement.
Theater-flag prompt variants
| Prompt | Precision | Recall | F1 | Parse failures |
|---|---|---|---|---|
| Baseline prompt | 75.45% | 78.06% | 0.7673 | 0 |
| No rationale | 77.64% | 79.37% | 0.7849 | 19 |
| Strict theater-intent rule | 77.40% | 79.87% | 0.7862 | 23 |
| Strict prefill, 80-token cap | 78.26% | 80.00% | 0.7912 | 20 |
| Strict prefill, 180-token cap · selected | 77.98% | 79.94% | 0.7895 | 0 |
Build a repeatable labeling job
Start with a label schema and reviewed examples that distinguish sentiment from the business question you need to answer. Run the open model over the table, validate each response, and retry malformed outputs separately. Compare positives and disagreements with a frontier model or a human reviewer, alongside a random control sample.
Save the reviewed rows as evaluation cases. Use recurring errors to revise the prompt or train the model, then score the next candidate on the same cases before replacing the current version. Track review calls, retries, and warehouse compute alongside model spend to budget each refresh.
For teams that need inference inside Snowflake, the proposed deployment imports the open weights and runs the model alongside the table. Define the domain, labels, examples, and evaluation rules first; public or synthetic examples can support model development.