One table, three labelers.
The source table contains 39,962 non-empty YouTube comments in a Snowflake-backed warehouse. All three model routes labeled the same table for broad sentiment, intent and a sparse business-specific label: explicit theater-attendance intent.
The original publication describes the candidate as a post-trained 30B open model. Exact model revision, training recipe, inference date, hardware and serving configuration are not reported in this public snapshot; Sonnet and Opus are published without exact model-version pins.
Full-table comparison
| Route | Rows | Parse failures | Positive labels | Reported cost |
|---|---|---|---|---|
| Post-trained 30B open route | 39,962 | 13 | 368 | $2.81589975 |
| Sonnet | 39,962 | 0 | 360 | $12.483859 |
| Opus | 39,962 | 1 | 405 | $139.63146 |
High agreement can hide a difficult positive class.
On the 39,948 rows valid across all three routes, three-way agreement was 69.55% for sentiment, 69.17% for intent and 99.30% for the theater-intent flag. These are model-agreement measures, not accuracy against human ground truth.
The theater flag is positive on only about one percent of the full table. Mostly agreeing on negative rows can create a high overall percentage while the positive class remains uncertain.
The checked-in publication data reports 319 strict-consensus positives. The open route hit 249 of them: recall versus that consensus is 78.06%. It also marked 119 rows outside that consensus, giving 67.66% precision versus consensus. Consensus is itself a model-derived reference, not a reviewed answer key.
Pairwise agreement on valid overlap
| Pair | Valid rows | Sentiment | Intent | Theater flag |
|---|---|---|---|---|
| Sonnet vs Opus | 39,961 | 85.92% | 81.71% | 99.68% |
| Sonnet vs open route | 39,949 | 72.88% | 76.69% | 99.54% |
| Opus vs open route | 39,948 | 77.62% | 77.76% | 99.39% |
A cost result, with an explicit quality boundary.
The lower-cost route makes disagreement review and selective reruns more affordable. Whether it clears a production quality bar still depends on reviewed examples, the cost of false positives and false negatives, and the output-validity requirement.
The public page does not report a human-labeled accuracy set, replicate uncertainty, model pricing date, or a fully loaded warehouse-compute cost. The quoted dollar figures are the published full-table model costs, not an assurance of the same price on another workload or deployment.
This presentation preserves the original measurements and clarifies rounded costs and agreement definitions. It is an editorial reconstruction of the public snapshot, not a new experiment or a ground-truth adjudication.
Limitations and source record.
- Three-way model agreement is not accuracy; no human ground-truth accuracy is reported.
- The rare positive label needs precision and recall analysis, not overall agreement alone.
- 13 open-route parse failures mean that attempted full-table coverage is not perfect valid-output coverage.
- Model-version pins, run date, confidence intervals and fully loaded warehouse costs are not reported in the public snapshot.
Source: the original Understudy publication and its checked-in benchmark snapshot. Adapted for Orchestra on 5 September 2026. This page preserves historical evidence; it is not a new run, live telemetry or a forecast.
Read the original publication ↗