Rapid LLM development is outrunning the research meant to explain it, study finds

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found that 44 percent have already "expired." Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months. The two examined the claims using models released between July 2024 and September 2026 by eight different developers. The original effect behind the expired findings has largely disappeared, while another 34 percent have weakened significantly.

Only 16 percent still hold, and 6 percent have actually gotten stronger. After one year, 16 percent had expired. After two years, that number climbed to 39 percent. Claims about what models can't do expire fastest, according to the study. Five of nine reasoning-related findings are considered expired, and none have held up or gotten stronger. They include well-known weaknesses like counting letters in "strawberry," arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and giving up correct answers when users push back.

The researchers suspect these kinds of errors have a…