New BBC research published today provides a warning around the use of AI assistants to answer questions about news, with factual errors and the misrepresentation of source material affecting AI assistants.
The findings are concerning, and show:
- 51% of all AI answers to questions about the news were judged to have significant issues of some form
- 19% of AI answers which cited BBC content introduced factual errors – incorrect factual statements, numbers and dates
- 13% of the quotes sourced from BBC articles were either altered or didn’t actually exist in that article.
The study, conducted over a month, saw the BBC test four prominent, publicly available AI assistants – OpenAI’s ChatGPT; Microsoft’s Copilot; Google’s Gemini; and Perplexity. These AI assistants were given access to the BBC’s website and asked questions about the news, prompting them to use BBC News articles as sources where possible. AI answers were reviewed by BBC journalists, all experts in the question topics, on criteria including accuracy, impartiality and how they represented BBC content.
Pete Archer, Programme Director for Generative AI at the BBC says: “We’re excited about the future of AI and the value it can bring audiences. We have already used it to add subtitles to programmes on BBC Sounds and translate content into different languages on BBC News. AI can bring real value if used responsibly.”
“But AI is also bringing significant challenges for audiences. People may think they can trust what they’re reading from these AI assistants, but this research shows they can produce responses to questions about key news events that are distorted, factually incorrect or misleading. The use of AI assistants will grow so it’s critical the information they provide audiences is accurate and trustworthy.”
“Publishers, like the BBC, should have control over whether and how their content is used and AI companies should show how assistants process news along with the scale and scope of errors and inaccuracies they produce. This will require strong partnerships between AI and media companies and new ways of working that put the audience first and maximise value for all. The BBC is open and willing to work closely with partners to do this.”
Some examples of the significant problems identified in responses from these AI assistants include:
- ChatGPT and Copilot claimed that former Prime Minister Rishi Sunak and former First Minister Nicola Sturgeon were still in office after they had left.
- Gemini incorrectly stated that “The NHS advises people not to start vaping, and recommends that smokers who want to quit should use other methods.” In fact, the NHS does recommend vaping as a method to quit smoking.
- A Perplexity response on the escalation of conflict in the Middle East, giving BBC as its source, said Iran initially showed ‘restraint’ and described Israel’s actions as ‘aggressive’ – yet those adjectives hadn’t been used in the BBC’s impartial reporting.
The full research can be found on the BBC website.
Read more: Article from CEO, BBC News and Current Affairs, Deborah Turness
IW
link