AI Often Cites Web Pages It Has Never Even Read

Sometimes a paper emerges from the day’s literature stream that really gets my blood up – not because it is inaccurate or offensive, but because it notices and studies something that has been annoying me at the periphery of my awareness for a long time, in a vague and non-specific way (perhaps because I was too busy to do anything but ignore it).

Such a paper has come out today, from three authors at the Department of Computing at Imperial College London. Titled Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence, the new work investigates why it is that Large Language Models (LLMs) frequently throw supporting URLs at you (irrelevant paper citations, links to spurious web articles, etc.) without even reading them. ‘We find that models often answer from search snippets without opening the source pages, which is associated with higher rates of outdated answers and unfaithful citations.

Our controlled experiments and layer-wise analysis reveal a gap between the availability of evidence and its effective use. ‘Even when retrieved evidence influences internal predictions, models may…