All research

Finding story boundaries in TV news bulletins: LLMs vs TextTiling

Result

Gemini 4 Argon on the transcript reaches 86.7 F1 (within 15 s), inside the 87.8 agreement ceiling. The best open model, Gemma 4 26B, is 5.8 points behind; embedding TextTiling trails by 20 points or more, even when told the number of stories.

Data
21 bulletins
Compute
Hosted APIs
Truth
Silver
Metric
Boundary F1

A 30-minute bulletin is a dozen stories glued together, and a short should be one of them. So the pipeline's first editorial step is finding where each story starts. The classic answer is TextTiling: embed the transcript and cut where the topic shifts. On 21 real bulletins it finds about half the boundaries. An LLM reading the same transcript matches the reference about as well as the reference matches itself.

Read the full study on Every Local AI