All research

Labelling news bulletin segments: can cheap models match a frontier one?

Result

Gemini 4 Argon agrees with itself on 88.6% of cells; Gemini 3.5 Flash and Flash-Lite reach 71-74%. Open models without video trail further (56-65%): telling a soundbite from a package needs to see who is on screen.

Data
21 bulletins
Compute
Hosted APIs
Truth
Silver
Metric
Macro-F1

Once a bulletin is split into stories, the pipeline needs to know what each stretch is: the anchor reading, a pre-recorded package, a soundbite from someone in the story, an interview, the opening titles. Soundbites especially matter, because a short is usually built around one. A frontier video model labels this consistently. The question was whether a cheaper one could do the same job. Not yet: the cheap models land about 15 points below the frontier model's agreement with itself.

Read the full study on Every Local AI