The Compute Bottleneck Intuition Is Formed Under Unusual Conditions

I often talk to AI researchers at frontier labs who think that very fast recursive self-improvement (e.g. a software-only intelligence explosion) isn’t possible, because it’ll be bottlenecked by experimental compute. This makes total sense—in our day-to-day experience, compute is often what we’re waiting on.

I share this intuition, but I think it’s misleading when applied to AI recursive self-improvement in a key way: humans can’t meaningfully increase our intelligence at all (in the relevant sense).

People do get smarter over time, of course: we get more experience, we develop tacit knowledge, and crystallized intelligence is real. But fluid intelligence is pretty consistent across people’s lives, and in the scope of all animals/potential future AIs, we’re not really able to change our intelligence very substantially. The kinds of improvements in intelligence that we’ve seen from GPT-2 to GPT-4 and GPT-4 to recent models are not available to humans, unfortunately. But every new generation of AI systems over the past 7 years has been meaningfully smarter than the previous one, and most indicators tell us we should expect that to continue.

So this intuition we have about how we’re bottlenecked on experimental compute vs. intelligence is formed under conditions that won’t generalize, because we’re each basically operating with a fixed level of intelligence. So, because we can’t really change our intelligence very much, it would be a waste of time to think about how sensitive our research output might be to it. On the other hand, because more compute is often attainable, it makes sense that we frequently think about it as a bottleneck.

Even within the relatively narrow range of human intelligence, research ability varies enormously¹E.g. Lotka’s Law describes how a small fraction of scientists produce the bulk of published output., suggesting output is very sensitive to intelligence even before we consider the much larger jumps AI systems have been making.

Of course, whether a software intelligence explosion happens is a very complicated/difficult question to answer²See Forethought’s Will AI R&D Automation Cause a Software Intelligence Explosion?, Epoch’s The software intelligence explosion debate needs experiments, and How fast can algorithms advance capabilities?., and considerations point in different directions. This is by no means a knockdown argument! But I think it’s an important and underrated consideration I want more people to consider explicitly.