AIs are past the point where they can now be reliably constrained by human safety guardrails1. This implies grave future risks in bioweapons and cybersecurity, as well as for the future of warfare.
We must be investing a lot more in improving our cyber defences than we currently do, at all levels. At the individual level, most people's online infrastructure, including their key email and social media accounts (hence their online data), is highly vulnerable to an attack (including from mistakes). Re-using passwords, and keeping app permissions ongoing for months, are big no-nos. At a minimum, you must use biometric passkeys and two-factor authentication. Google lets you run a security checkup on any accounts you have with it. They also offer an advanced protection program, and web-searching protection, completely free.
In theory, I am a lot more sympathetic than I was yesterday to pausing frontier AI training and development for the rest of the year, until we get to grips with misaligned models. Yet some open source is on par, and a pause benefits them, as they would continue to improve capabilities. Unless we can coordinate an international slowdown across all the major powers, then RSI is coming imminently, and there is not much anyone can do about it.
My predictions (far from superforecaster level of rigour but worth saying):
1. Conditional on a future AI-assisted bioweapon or pandemic happening, the magnitude of such will be comparable to that of the Covid pandemic in its social costs.
2. Conditional on a major cybersecurity incident occuring (attacking crucial national infrastructure), the cumulative costs might also be equivalent to a pandemic (or even a small war).
3. Wars already occuring are due to get a lot more deadly. The probability of conflict may also increase.
4. Government responses to 1) or 2) may well be worse than the actual shock itself. Markets and innovation have already produced rather robust decentralised security tools.
5. I think there is a better-than-even chance of 1) or 2) occuring within my lifetime (note: I expect to live at least well into my 100s due to AI-driven biotech advances too).
Humanity will not only survive nonetheless, but the singularity will still take hold, and we will still be a lot more prosperous. The Great Stagnation was unlikely to last, and growth dynamics fit an exponential curve well (albeit there exist nonlinearities in progress in means that endogenous growth models don't fully capture). This matches the Whig view of inevitable historical progress, and so applies to our institutions just as much as capital technology too. Today will be akin to the Middle Ages before the Industrial Revolution.
N.B. Elsewhere, I have reported my p(doom)=(5%,6%) by 2070, which is within the norm, and this has not changed. I think there is a much higher chance of AI causing a major disastrous event, yet not on a scale large enough to wipe out humanity as we know it. Arguably, the post-Roman Dark Ages in Europe is close to what many consider a catastrophe to be (minus total extinction). We survived that, and still ended up in a singularity. My view of progress is exponential yet nonlinear, so suppose that AI does indeed cause a catastrophe - it is not necessarily permanent doom. This p(doom) is still high enough for this to be the top issue that we should be focusing on today, by far.
AI 2027 has a somewhat doomer reputation, but it forecasts a mostly positive ending. Their optimistic scenario, outlined in further detail in AI 2040, is indeed conditional on an international treaty, yet we did this with nuclear weapons at the height of the Cold War, so I am optimistic of our chances of making an agreement with China (and by extension Russia, whom has to be aligned with China to even sustain its military capabilities). Our world will look a lot different yet more prosperous in ways we're only starting to imagine in a few years.
N.B.2. One significant point of disagreement I have with AI 2027 is they underestimate lags and bottlenecks to adoption and diffusion across many sectors. They're on point on capabilities (Moore's Law really is an invariant trend where extrapolation is sufficient - no regime change or Lucas Critique applies here), not less so on the social and economic impact.
My median date for robots outplacing humans in factories is 2042. 10-15 years for diffusion (a date sorta midpoint of this range), and (with this announcement) three or so years for today's robots to match Fable or Mythos in capabilities. It's plausible that we'll cure most forms of cancer earlier, with the advent of personalised mRNA vaccines. We pretty much cured Covid years after those vaccines went through the clinical trials. If someone can point me to a GJ or Metaculus question on a similar subject, I'll do a forecast now.
My median assessment for growth in the 2030s is around 3-4% (although an optimistic scenario adds an extra percentage point), with another percentage point in the 2040s (when AI is comparable to the second Industrial Revolution). I also think, given the r>g dynamics in Halperin et al 2026, that AI could well cause a fiscal crisis before the singularity. Nonetheless, here are my predictions on record. If Britain legalised prediction markets, I'd bet on this.
This does not necessarily imply that the constraints imply zero effect, but nonetheless the number of these security incidents are above their optimal number.