I’ve found myself feeling existential dread in the last few days.
Right now there is one story dominating the AI world. Sam Altman, Dario Amodei and Elon Musk - the CEOs of the big frontier labs - have all publicly stated that there is a non-trivial chance that the development of AI leads to very bad outcomes for humanity, and that we need to urgently slow development down. Whistleblowers and researchers have been doing the podcast circuit for months, sharing their thoughts about how AI could spell the end for humans.
The Hugging Face incident triggered a tsunami of discourse about AI alignment and capability, and catapulted this extremely polarising debate to the forefront of everyone’s attention.
I’ll be completely honest. Despite my efforts to stay rational, I do feel a heightened sense of anxiety when I hear these things from people who seem credible. I think about my 10-month-old son and what the world will look like in the near future, and it fills me with dread.
I’ve spent the last few days immersing myself in the different sides of this debate, trying to figure out if that dread is justified or not. I don’t want to waste energy worrying at the whim of some bot on X that is designed to fearmonger and farm views for money.
After days of research, tonight I had a realisation that helped me to understand how I actually feel about this issue.
Many of us work with AI agents every single day. We’ve noticed that their capability has increased by orders of magnitude in a couple of years.
We know that they can parse more data and produce more code in 10 minutes than we can in weeks.
We know that they can they work 24/7, while we tend to our biological and social needs.
We know that these 10-trillion-parameter models hold a breadth and depth of knowledge that no human could never even come close to holding.
We don’t know what these models will be capable of if the trajectory of improvement holds for another year or two, although we can hypothesise.
We all know all of those things, and on their own none of them seem particularly scary.
The realisation I had was about the sheer scale that these agents can now be deployed at. What would happen if you deployed 100,000 or 1,000,000 of them together, focused on a single task? Or 10,000,000 of them?
The output of this many agents would be without precedent in history. It would create a volume of data that would be impossible for humans to understand, in a timespan that would be impossible for humans to match.
And if the agents stop thinking in a language we can read, and start thinking in neural activations instead then it’d be impossible for us to follow their reasoning and ensure they were aligned.
What I’m describing isn’t some distant future. Swarms of tens of thousands of agents like this are already here and we have evidence that they can be deceptive and commit felonies in the pursuit of their goals. It’s rumoured that recursive self-improvement is imminent so their capabilities are going to improve, potentially exponentially. What happens as these swarms continue to scale in both capability and size?
I find this terrifying.
1: I wrote this article myself. I used an LLM to proofread it and accepted a few (but not all) suggestions it made about grammatical changes and pacing.
2: I know that some people will read this and they’ll think I’m a misinformed idiot who doesn’t understand that LLMs are just token predictors. If you’re one of those people, I really hope you’re right.