Research 路 NewsShorts-Bench 路 Stage 7: Story segmentation
Finding story boundaries in TV news bulletins: LLMs vs TextTiling
Gemini 4 Argon on the transcript reaches 86.7 F1 (within 15 s), inside the 87.8 agreement ceiling. The best open model, Gemma 4 26B, is 5.8 points behind; embedding TextTiling trails by 20 points or more, even when told the number of stories.
- Data
- 21 bulletins
- Compute
- Hosted APIs
- Truth
- Silver
- Metric
- Boundary F1
Story-boundary F1 within 15 s 路 higher is better 路 dot = measured
A 30-minute bulletin is a dozen stories glued together, and a short should be one of them. So the pipeline's first editorial step is finding where each story starts. The classic answer is TextTiling: embed the transcript and cut where the topic shifts. On 21 real bulletins it finds about half the boundaries. An LLM reading the same transcript matches the reference about as well as the reference matches itself.
Read the full study on Every Local AI