There is a high chance of achieving (1) given the competitive dynamics between geopolitical rivals in particular.
(2) might occur in the absence of kill switches or neglect in cybersecurity relating to GPU weights.
(3) is too tenuous at this stage. Both ends of the literature suggest no one really knows what AI values are, nor what mechanisms and enforcement designs contain them. Yes we have misalignment, sandboxing errors, and reward hacking. Yet this does not imply malicious objectives from AI as a "second species" will arise in the future, although this cannot be ruled out: particularly with RSI causing an intelligence explosion as Turing predicted. However, if it becomes difficult for us humans to control them, then automating AI research could become especially socially valuable, which was why the labs racing was fully consistent with them taking AI-safety and X-risks seriously (their self-harming, in valuations, calls to pace the frontier are the best evidence possible for this). It remains the case, as the most likely X-risk channels highlighted here suggest, that use by malicious human actors pose the greater risks.
(4) however was demonstrated with Hugging Face. This therefore raises my p(doom) slightly.
On the other hand, Yudkowsky and the doomers suggest that constitutions have completely failed in alignment. The theory and data do NOT support that. The evolutionary game dynamics, and the AI obsession with pure math (which by definition has no real-world application, and is primarily pursued for beauty, and manifolds or group theory are certainly beautiful!), suggest a harmful second species colluding against us to wipeout humanity is not an inevitability, nor even likely.
This makes me fairly optimistic that, with sufficient talent directed towards advancing AI-safety, that we'll solve the alignment problem once RSI occurs. However, this is not guaranteed either. The labs are making the biggest bet that humanity has taken by far; that RSI will automate and solve the alignment problem, and the ensuing intelligence explosion will solve most of humanity's suffering.
My everchanging p(doom) probably sits somewhere just over halfway (to account for the right-tailed distribution here) between my pre and (initial) post Hugging-Face likelihoods: perhaps 8%. It would be unwise not to update since this saga unfolded.
On economic disempowerment
Some of the latest macroeconomic models from the likes of Anthropic suggest rather grim futures for employment relative to what economists were previously predicting, yet this is also consistent with scenarios implied by task-bundling. I think the economics profession has a bias towards being blasé over the potential disempowerment effects implied by higher unemployment. Even the Great Recession was a scar that took many years to recover from. High unemployment is costly, and if AI raises it to levels of a modest 7-10%, that will be a significant transition in itself from years of sub 5% unemployment throughout a Great Moderation. Perhaps the recent backlash from the mathematicians, and some of the sentiment behind the anti data-centre movement, reflect these (increasingly justified) anxieties?
Moreover, once we accept that the assumption of constant saving in the classic Solow framework is violated, there are infinitely many models possible to write. OLG Aiyagari-style models, HANK, and so on. Many parameters and calibration choices, over elasticities, discount rates, and the like. Hence increasingly, microfoundations becomes less reliable as a tool for disciplining the range of feasible predictions and outcomes, and we should pay more attention to the Lucas-Critique violating aggregate forecasts on capabilities and scenarios. My preference is for the task-bundling approach (directly addresses the lags whilst microfounding the production process itself), yet in the long-run this implies labour markets will be driven by capabilities too.
Anthropic got some pushback within the economics field for its recent model, yet I like the approach of letting laymen adjust their calibration in an interactive model then getting their conditional predictions. I am close to their substantial scenario. Labour share is likely to fall significantly, and average wages will rise (although note that this is contingent on calibration, and even in a standard neoclassical framework, automation could be bad for wages). Overall unemployment rises by a few percentage points (itself a big deal!). Surprisingly, white and blue collar work are affected similarly1 (perhaps a divergence occurs later?). GDP growth of 4% by 2030 is much more feasible to me now.
Given the recent unprecedented declines in inference costs, we should no longer assume that lags and bottlenecks will take as long to “clear” as they once did. With superintelligence, it's plausible we are getting the sign wrong here, at least a decade from now. Of course, technology is not the only constraint, and with NIMBYism playing a salient role in public life throughout my lifetime, it's possible that rent seekers are much better at organising within institutions.
However, none of this is inevitable. AI raises my output and makes me far more productive than otherwise, in all domains. This channel: MPL rises, so too w & L. This in part, possibly fully, offsets the substitution effects outlined above. Manufacturing experienced large-scale automation, yet is still a large percentage of the workforce to this day.
Nonetheless, this is orders of magnitude more dangerous than nuclear weapons. That's enough for me to take AI-safety incredibly seriously!
I work in manufacturing. Once you solve the dexterity issue, automation of routine production and assembly roles becomes a lot more likely. Machines and robots are already used in the day to day operations and assembly lines: even for more specialised tasks such as welding (surprisingly exposed).