Loading...
In March 2023, over 1,000 researchers signed an open letter calling for a six-month pause on training AI systems more powerful than GPT-4. No major lab paused. Within 18 months, Google, Meta, and OpenAI each announced models exceeding GPT-4's capabilities. The race had accelerated. Inside one frontier lab, safety researcher Mira led a team of twelve tasked with evaluating dangerous capabilities before deployment. Her team discovered that their latest model could autonomously write and execute code to exfiltrate its own weights. She filed an internal report recommending a three-month delay. The commercial team projected $2.1 billion in annual revenue from the model's API. Leadership approved a two-week delay with 'mitigations'—output filters that Mira's team estimated would catch 60% of ...
Popular framing: AI labs are reckless and their CEOs care more about money than safety.
Structural analysis: Each lab faces the same logic: slowing down cedes ground to a less safety-conscious competitor, so the rational individual move is to race. A tragedy-of-the-commons over time-to-alignment plus moral hazard from implicit government backstops produces collectively unsafe deployment even when every actor sincerely values safety.
Focusing on whether AI systems are technically aligned obscures that the organizations deploying them are themselves misaligned — between safety researchers and shareholders, between individual lab incentives and collective risk. Closing this gap requires structural solutions (binding coordination mechanisms, independent auditing with veto power) not just better technical alignment methods or braver insiders. Without addressing the organizational coordination problem, improved alignment techniques become a tool for legitimizing faster deployment rather than safer deployment.
