Building standards for the next phase of AI

PUBLISHER ARTICLE PREVIEW

From the original article

Our mission is to ensure that artificial general intelligence benefits all of humanity. As outlined recently by Sam Altman and Jakub Pachocki, we prioritize our work towards this mission with three main goals:

Navigate the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem, and finding ways for people to remain part of the self-improvement loop.

Deliver the benefits of scientific progress and economic growth that very intelligent machines enable.

AI is accelerating our own research and engineering, and AI-enabled research has led to advances in mathematics, including the Navier-Stokes Millennium Problem. This progress holds great promise for our other two goals, which could lead to important discoveries in medicine, broadly accessible health care, and economic empowerment for small businesses and individual entrepreneurs everywhere.

AI is advancing across many companies and countries, and progress across the field changes the opportunities and challenges facing everyone. Alongside its other benefits, capable, aligned AI will be an important part of how we navigate this transition—strengthening defenses, advancing alignment research, and helping people understand and guide the next stages of AI development.

Navigating this transition safely requires alignment research to keep pace with these capabilities so that the systems we and others build remain aligned with human values and under human control. This becomes increasingly important as AI systems grow more capable and autonomous, including as they take on more of AI research and development itself. Making continued AI progress safe and beneficial will require both advances in alignment research and shared standards to guide development across labs and countries.

Automated AI research can involve varying degrees of human supervision. As AI systems take on more of the work of developing successive generations of AI, they can increasingly drive a process of recursive self-improvement (RSI), even while people remain involved. As this process becomes more automated, the pace of AI progress could accelerate rapidly.

We believe automated AI research will yield models that directly enhance people’s lives. It can bring down the cost of advanced intelligence so that people worldwide can benefit. Automated research could also help us substantially improve alignment and build defenses against increasingly capable AI—an automated AI researcher can also be an automated AI safety researcher.

More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.

Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.

From here, AI could become more dangerous, less aligned, and, on the