Research · NewsShorts-Bench · Stage 9: Highlight selection
Which news stories make good shorts? LLM rankings vs their own ceiling
Story choice is subjective: Gemini 4 Argon agrees with itself at Spearman 0.70 and picks the same top story 58% of the time. Open models on transcripts rank as well as Gemini 3.5 Flash, but where to cut needs video (clip IoU 0.65-0.77 vs 0.43-0.45).
- Data
- 174 stories
- Compute
- Hosted APIs
- Truth
- Silver
- Metric
- Spearman
Rank correlation with the reference (Spearman) · higher is better · dot = measured · bar = 95% CI
Which story from tonight's bulletin should become a short, and which 40 seconds of it? Every "AI clipping" tool claims to answer this. We asked four models to rank 174 real stories and pick the cut, and checked first how much the reference model agrees with itself. It agrees only so-so. The same frontier model, asked with different wording, picks the same top story 58% of the time. Story choice is a judgment call. Where to cut is not: that part needs the video.
Read the full study on Every Local AI