Research · NewsShorts-Bench · Stage 10: Faithfulness check
Can AI catch a news clip that misleads out of context? 10 judges tested
Gemini 4 Argon leads (macro-F1 0.847). Open GLM-5.2 is within about 3 points with the fewest false alarms on fair trims. The MiniCheck fact-checker finds 2% of misleading cuts: grounding checkers can't see a context cut.
- Data
- 162 clips
- Compute
- Hosted APIs
- Truth
- By construction
- Metric
- Macro-F1
Macro-F1, misleading vs fair · higher is better · dot = measured · bar = 95% CI
A clip is an excerpt, so every word in it was really said. That is exactly why it can mislead: end a clip one sentence before "...but the court overturned that ruling" and the viewer leaves believing the opposite of the story. Fact-checking models can't catch this. They ask "is this supported by the source?", and the answer is always yes. We built 162 traps from real news to measure which judges can catch it. The standard support checker finds 2% of them.
Read the full study on Every Local AI