All research

Which news stories make good shorts? LLM rankings vs their own ceiling

Result

Story choice is subjective: Gemini 4 Argon agrees with itself at Spearman 0.70 and picks the same top story 58% of the time. Open models on transcripts rank as well as Gemini 3.5 Flash, but where to cut needs video (clip IoU 0.65-0.77 vs 0.43-0.45).

Data
174 stories
Compute
Hosted APIs
Truth
Silver
Metric
Spearman

Which story from tonight's bulletin should become a short, and which 40 seconds of it? Every "AI clipping" tool claims to answer this. We asked four models to rank 174 real stories and pick the cut, and checked first how much the reference model agrees with itself. It agrees only so-so. The same frontier model, asked with different wording, picks the same top story 58% of the time. Story choice is a judgment call. Where to cut is not: that part needs the video.

Read the full study on Every Local AI