AI Summaries Get News Wrong

February 28, 2025

With big news stories emerging at a frantic pace, one might turn to AI to consolidate the key points. If so, one might become woefully ill informed. “AI Chatbots Unable to Accurately Summarise News, BBC Finds.” The BBC tested the biggest AIs on content from its own site–OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini and Perplexity AI all sat for the exam. None of them passed it, though ChatGPT and Perplexity were less bad than Copilot and Gemini. Tech reporter Imran Rahman-Jones tells us:

“In the study, the BBC asked ChatGPT, Copilot, Gemini and Perplexity to summarise 100 news stories and rated each answer. It got journalists who were relevant experts in the subject of the article to rate the quality of answers from the AI assistants. It found 51% of all AI answers to questions about the news were judged to have significant issues of some form. Additionally, 19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates.”

But it was not just about mixing up, or inventing, facts. The chatbots also struggled with the concept of context and the distinction between facts and opinions. We learn:

“The report said that as well as containing factual inaccuracies, the chatbots ‘struggled to differentiate between opinion and fact, editorialised, and often failed to include essential context’.”

To illustrate the findings, the article gives us a few examples:

“Gemini incorrectly said the NHS did not recommend vaping as an aid to quit smoking.
ChatGPT and Copilot said Rishi Sunak and Nicola Sturgeon were still in office even after they had left.
Perplexity misquoted BBC News in a story about the Middle East, saying Iran initially showed ‘restraint’ and described Israel’s actions as ‘aggressive’.”

So, dear readers, we suggest you take the time to read the news for yourselves. Or, at the very least, get your recaps from another human.

Cynthia Murrell, February 28, 2025

Written by Stephen E. Arnold · Filed Under AI, News

Comments

Got something to say?

Search the site
Subscribe to Beyond Search
Feature archive
News archive

Stephen E. Arnold monitors search, content processing, text mining and related topics from his high-tech nerve center in rural Kentucky. He tries to winnow the goose feathers from the giblets. He works with colleagues worldwide to make this Web log useful to those who want to go "beyond search". Contact him at sa [at] arnoldit.com. His Web site with additional information about search is arnoldit.com.