Research · NewsShorts-Bench · Stage 8: Segment roles
Labelling news bulletin segments: can cheap models match a frontier one?
Gemini 4 Argon agrees with itself on 88.6% of cells; Gemini 3.5 Flash and Flash-Lite reach 71-74%. Open models without video trail further (56-65%): telling a soundbite from a package needs to see who is on screen.
- Data
- 21 bulletins
- Compute
- Hosted APIs
- Truth
- Silver
- Metric
- Macro-F1
Macro-F1 over segment roles (%) · higher is better · dot = measured · bar = 95% CI
Once a bulletin is split into stories, the pipeline needs to know what each stretch is: the anchor reading, a pre-recorded package, a soundbite from someone in the story, an interview, the opening titles. Soundbites especially matter, because a short is usually built around one. A frontier video model labels this consistently. The question was whether a cheaper one could do the same job. Not yet: the cheap models land about 15 points below the frontier model's agreement with itself.
Read the full study on Every Local AI