Giving AI Video Some Remembrance of Things Past

PUBLISHER ARTICLE PREVIEW

From the original article

Back in March we looked at the difficulty generative AI video has in remembering things and places that it has seen, so that it can reproduce them a second time, and offer visual consistency.

The net effect of the problem is that you may walk through a door in an AI world model, turn round ten seconds later, and find that the door you came through is no longer there.

It’s not a trivial glitch to fix: since generative AI video is constantly inventing ‘the next thing’ it must contend with, it cannot really remember anything very far back, or anything that has expired from its narrow context window.

This means that if you explore a trained AI world model, it’s not like exploring a modeled CGI environment, where the textures and geometry are fixed, and saved as vector and bitmapped data; instead, it’s like walking through a dream of approximate probabilities – you may find the exact same thing twice, but that’s only because it was summoned up a second time from the latent space via context; not because the model actually remembers creating it or seeing it.

Thus, the average genAI video world model is similar to someone with anterograde amnesia, who has total recall prior to a cerebral event – such as a stroke or injury – but who cannot make new memories since the incident (a condition given publicity by Guy Pearce’s character suffering from it in Christopher Nolan’s Memento [2000], featured in the main article image).

In this way, video models retain a deep and profoundly-embedded understanding of many domains; but without the capability to remember anything new.

The HyDRA solution that we looked at in March was based on the use of a ‘memory bank’ of compressed visual tokens that would allow AI-generated characters to move offscreen and return in the same guise as when they were first seen.

There are a number of other methods that use this ‘breadcrumb’ approach; however, generating and leaving behind complex 3D geometry solves the problem at the expense of file size, as well as responsiveness, latency, and other potential bottlenecks.

These ‘3D-ground world models’ include the NVIDIA-led Gen3C, whose HyDRA approach uses the most typical method of resolving this particular problem – turning what it ‘sees’ into point-clouds, which are essentially stripped-down 3D geometry:…

Click to play. The HyDRA approach uses a persistent 3D point-cloud cache to guide video generation, preserving scene consistency as the camera moves and previously seen areas reappear.

There are other similar approaches, such as Oxford University’s 2025 VMEM, and the Stanford-led SPMem collaboration from the same period:…

Click to play. From the Stanford-led collaboration SPMem, which combines a persistent 3D map of static scenery with recent video frames for motion, and selected older frames for visual detail, helping the model remember places when revisiting them.

But all 3D geometry is heavy work, and hard on inference requirements. It’s not easy to walk round an environment and be constantly ‘building’ it as a CG artist might. To boot, any other clips or viewpoints or takes will have to be able to reference the created geometry later, which is a challenging proposition.