Is this the end?
We are in the singularity, so humanity will not just survive but thrive.
AIs are past the point where they can now be reliably constrained by human safety guardrails1. This implies grave future risks in bioweapons and cybersecurity, as well as for the future of warfare.
We must be investing a lot more in improving our cyber defences than we currently do, at all levels. At the individual level, most people's online infrastructure, including their key email and social media accounts (hence their online data), is highly vulnerable to an attack (including from mistakes). Re-using passwords, and keeping app permissions ongoing for months, are big no-nos. At a minimum, you must use biometric passkeys and two-factor authentication. Google lets you run a security checkup on any accounts you have with it. They also offer an advanced protection program, and web-searching protection, completely free.
In theory, I am a lot more sympathetic than I was yesterday to pausing frontier AI training and development for the rest of the year, until we get to grips with misaligned models. Yet some open source is on par, and a pause benefits them, as they would continue to improve capabilities.
My predictions (far from superforecaster level of rigour but worth saying):
1. Conditional on a future AI-assisted bioweapon or pandemic happening, the magnitude of such will be comparable to that of the Covid pandemic in its social costs.
2. Conditional on a major cybersecurity incident occuring (attacking crucial national infrastructure), the cumulative costs might also be equivalent to a pandemic (or even a small war).
3. Wars already occuring are due to get a lot more deadly. The probability of conflict may also increase.
4. Government responses to 1) or 2) may well be worse than the actual shock itself. Markets and innovation have already produced rather robust decentralised security tools.
5. I think there is a better-than-even chance of 1) or 2) occuring within my lifetime (note: I expect to live at least well into my 100s due to AI-driven biotech advances too).
Humanity will not only survive nonetheless, but the singularity will still take hold, and we will still be a lot more prosperous. The Great Stagnation was unlikely to last, and growth dynamics fit an exponential curve well (albeit there exist nonlinearities in progress in means that endogenous growth models don't fully capture). This matches the Whig view of inevitable historical progress, and so applies to our institutions just as much as capital technology too. Today will be akin to the Middle Ages before the Industrial Revolution.
This does not necessarily imply that the constraints imply zero effect, but nonetheless the number of these security incidents are above their optimal number.