The AI Productivity Illusion

Productivity Is About (Much) More than Finishing Tasks Quickly

Matt Scherer is a fellow at Open Markets Institute, where his research and advocacy focus on developing policy responses to the eventual bursting of the AI bubble. His Hard Reset pieces focus on highlighting the risks posed by the AI bubble and pushing back against the hype that is inflating it. The opinions expressed here are solely his own.

It’s easy to see why people think generative AI is a revolutionary technology. After all, lots of jobs involve writing, and ChatGPT, Claude, and other large-language-model-based systems can write faster than any human can. As a result, and in addition to the anecdotes from people saying, “I have 10Xed my productivity using AI,” there are empirical studies showing that people can complete certain tasks, such as coding, significantly faster with AI. Because AI so clearly makes lots of individual people more productive, it seems like a foregone conclusion that it will do the same for the economy as a whole.

So why is it that, four years after ChatGPT’s release, generative AI has not yet improved productivity at either the level of individual companies or across the economy as a whole?

Some have referred to this apparent disconnect as a “productivity paradox,” a phrase that economist Erik Brynjolfsson (building on an observation by Robert Solow) coined to refer to the period in the 1970s and 1980s when information technology was advancing rapidly but economy-wide productivity statistics barely budged. Once companies restructured their organizations and workflows to center the computer and Internet, productivity growth did indeed pick up (although less dramatically than it did during the Industrial Revolution and after World War II). Brynjolfsson and others think the same thing is happening with AI. Once companies rewire themselves for the AI era, they say, real productivity gains will come.

But there may be a much simpler explanation: the mere fact that AI allows people to work faster does not mean it is increasing productivity, at least not in the ways that truly matter.

************************

In casual conversation, when someone says that they have increased their “productivity,” they typically mean they are producing more stuff in less time. In this context, productivity means:

But in the economic sense, which is the sense that matters in determining how a technology will affect businesses and economies, “productivity” (specifically, total factor productivity) is not about how much you produce or how quickly you produce it. It’s about the economic value of your inputs and outputs, rather like a large-scale version of return-on-investment:

To be sure, the colloquial and economic meanings of productivity overlap quite a bit. Looking at the numerators, more stuff typically translates to more economic value. A manufacturer can typically get more money by selling 1,000 widgets than it can by selling 500. Likewise, looking at the denominators, cutting down the time it takes to produce a good or service tends to increase economic productivity because time is a key economic input, especially if human labor is part of the production process. If you only need 10 hours to complete a $500 task that used to take you 20 hours, you have doubled both your casual/personal productivity and your true/economic productivity.

There is much more to economic productivity, however, than just quantity and speed. And that is where the disconnect between productivity in the more casual personal sense and productivity in the technical economic sense lies.

************************

For starters, the economic value of a good or service depends on its quality, not just its quantity. Quantity certainly matters; the Industrial Revolution began with textiles in large part because new machines made it possible to produce vastly more clothing and other textiles in vastly less time by automating the various tasks involved in textile production. But, crucially, automating those tasks did not reduce the quality of the textiles. In fact, it usually improved it.

A scene in the musical Fiddler on the Roof contains a fictional-but-memorable illustration of this. In the scene, the tailor Motel gets a new sewing machine. Motel excitedly explains to his neighbors that the sewing machine “works twice as fast.” Holding up a swath of fabric, he points out “how close and even the stitches are” and exclaims, “from now on, my clothes will be perfect—made by machine!”

It was not just the machine’s speed that he valued; it was its precision and reliability. Regardless of how quickly it allowed him to work, Motel presumably would have preferred to stick with manual tailoring if his sewing machine frequently went rogue and produced shirts that were the wrong size, contained a third sleeve, or unraveled in his customers’ hands.¹

************************

Which, of course, brings us to generative AI. Start with the denominator of the economic productivity equation. The AI infrastructure build-out has already cost $1.5 trillion, with spending expected to accelerate further next year, in addition to the perpetual costs associated with powering and cooling the chips. That means AI will have to generate a lot of value just to break even when it comes to economic productivity.

Supposedly, the way it will do so is by automating tasks in so-called “knowledge work” fields like software engineering, law, and medicine. But generative AI is notoriously and perhaps incurably prone to serious errors, such as fabricating case law, deleting code databases, and inserting incorrect patient diagnoses. The result — not unlike what Mickey encountered when he tried to get an enchanted broom to do his chores for him in The Sorcerer’s Apprentice scene from Fantasia — is often that professionals must spend more effort trying to clean up the AI’s messes than they would have spent if they had simply done the task themselves from the beginning.

Credit: IFA Film.

Such error-proneness hurts both parts of the economic productivity equation. In the numerator, output that contains serious errors or is otherwise low-quality will have little or no value. In the denominator, knowing that serious errors are a regular occurrence means that a significant amount of time must be devoted to quality control, which drives up input costs.

The assumption seems to be that such errors are, or soon will be, so rare that their negative impact on productivity will be swamped by the gains from increased output. But that is not how technological progress usually works. Despite tech bro promises that generative AI will get exponentially better, there usually are diminishing returns when it comes to squeezing additional improvements out of a given technology.

AI boosters are fond of saying that today’s AI is the worst AI you will ever use. But it’s just as likely that today’s generative AI is about as good as it will ever get.

************************

Even if generative AI’s reliability issues were miraculously solved, its outputs simply may not be economically valuable enough to significantly increase productivity. The Industrial Revolution was transformative because it made it much easier for people to get things they truly needed, sometimes quite desperately. Even if the production processes introduced some inefficiencies and flaws, the demand for clothing (and food, faster transportation, construction materials, etc) was so overwhelming that there was no question that dramatically increasing output would unlock immense economic value.

Will the same hold true for the things generative AI produces? The corporate world has been aggressively trying to automate, outsource, gigify, or eliminate office-based work for years, and much of the low-hanging fruit has already been plucked. Thus far, there is little evidence that AI is creating new economic value from the what’s left — even for tasks that generative AI is supposedly great at.

Recall this Financial Times chart from my last column:

AI has nearly doubled the number of app releases released per month. But overall, the number of apps with significant use has started to decline. The mere production of more stuff is not creating new economic value when it comes to app development. We shouldn’t be surprised if that pattern repeats in other white-collar fields where generative AI is still struggling to catch on.

Another force to bear in mind: many of the use cases where generative AI truly shines have a steeply negative effect on productivity. The rise of generative AI will almost certainly hurt the productivity of cybersecurity specialists because generative AI is great at launching cyberattacks but terrible at writing the secure code necessary to stop them. People are also using AI to hoodwink companies into hiring people for jobs they cannot perform. More subtly, low-quality AI “workslop” destroys productivity in many of the settings where it is used because the proliferation of such slop wastes time and makes it harder to identify work product that is fit-for-purpose.

While people talk about hallucinations and slop as if they are mere nuisances, they actually go to the core of why it’s doubtful AI can ever spur a new Industrial Revolution. Whatever gains generative AI unlocks from completing tasks faster may be swamped, or at least offset, by all the ways in which its unreliability destroys economic value.

With generative AI, there may not be a productivity paradox at all — just a productivity illusion.

1. Hat tip to my friend Owen Firestone for coming up with this line.