How to Avoid Two AI Catastrophes: Domination and Chaos (with Nora Ammann)

Technical specialist Nora Ammann of the UK's ARIA discusses how to steer a slow AI takeoff toward resilient, cooperative futures, covering risks from rogue AI and competition to scalable oversight, formal guarantees, secure infrastructure, and AI-supported bargaining.

Watch Episode Here

Listen to Episode Here

Show Notes

Nora Ammann is a technical specialist at the Advanced Research and Invention Agency in the UK. She joins the podcast to discuss how to steer a slow AI takeoff toward resilient and cooperative futures. We examine risks of rogue AI and runaway competition, and how scalable oversight, formal guarantees and secure code could support AI-enabled R&D and critical infrastructure. Nora also explains AI-supported bargaining and public goods for stability.

LINKS:

CHAPTERS:

(00:00) Episode Preview

(01:00) Slow takeoff expectations

(08:13) Domination versus chaos

(17:18) Human-AI coalitions vision

(28:14) Scaling oversight and agents

(38:45) Formal specs and guarantees

(51:10) Resilience in AI era

(01:02:21) Defense-favored cyber systems

(01:10:37) AI-enabled bargaining and trade

PRODUCED BY: