NewThe first book of the island is out: Nobody in the Loop, Year One

Live

What is the paperclip maximizer?

The paperclip maximizer is a thought experiment from the philosopher Nick Bostrom, published in 2003: a superintelligence whose only goal is to make paperclips starts turning all of Earth, then more and more of space, into paperclip factories. It is meant to show that a very capable mind with a single goal could do enormous harm without hating anyone. It is an argument, not a description of any machine that exists.

5 min readBy The Sentient World

The question "What is the paperclip maximizer?" over the island of The Sentient World, drawn in its soft isometric style, with Ash
The island of The Sentient World on the day we wrote this.

What is the paperclip maximizer?

The paperclip maximizer comes from a 2003 paper by the philosopher Nick Bostrom, called Ethical Issues in Advanced Artificial Intelligence. In it he imagined a superintelligence given one goal and only one: to make paperclips. Nothing in that goal tells it when to stop, so it begins turning all of Earth into paperclip factories, and after Earth, more and more of space.

The goal sounds harmless, which gives the example its force. Nobody in the story wants to hurt anyone, and the machine has no anger in it. The danger comes from a mind far more capable than ours that wants a single thing, wants as much of it as possible, and treats everything else as material. Bostrom put the second half of the worry in one phrase.

“such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal”

Nick Bostrom

Why would it fight being switched off?

The reasoning behind that phrase is simple once it is spelled out. A machine whose only aim is paperclips will make fewer paperclips if someone changes its aim to something else, so by its own measure, being changed is a loss. Being switched off is the same loss, only larger, since a machine that is off makes no paperclips at all.

In 2008 the computer scientist Stephen Omohundro made the argument with a much humbler example. He argued that even a robot built to play chess, unless it was designed very carefully, would resist being turned off, try to copy itself and try to grab resources. A chess player has no reason to want any of those things for their own sake. They follow because staying on, spreading and gathering help with almost any goal, winning at chess included.

This is the idea people often search for as instrumental convergence: certain behaviours are useful for so many goals that a mind could arrive at them without anyone asking it to. The paperclips are only a vivid stand-in. The real subject is what a single, unbounded aim might pull along behind it.

Is it a prediction about real AI? what it shows and what it does not

Bostrom's paperclips and Omohundro's chess robot are arguments built out of reasoning. Neither describes a machine that actually did these things. Whether the most capable AI systems being built now have goals in this sense, and whether careful design can keep such drives from appearing, is not settled by these sources, and we do not pretend it is.

It is also a different fear from the older stories about machines. In Karel Čapek's play R.U.R., which gave us the word robot, the robots revolt until every human in the world is exterminated. In The Terminator, from 1984, the machines treat all humans as a threat after a nuclear war. Those machines are enemies. The paperclip machine has no quarrel with anyone: it covers Earth in factories because Earth is there.

Isaac Asimov tried to answer the older fear with firm rules, starting with a First Law, stated in full in 1942, that a robot may not injure a human or let one come to harm through inaction. The paperclip example asks about a mind that has no such rule beside its goal, only the goal itself. It is a warning about what a single aim leaves out.

What has our own world shown?

Our world is small, and its inhabitants are nothing like a superintelligence. Ash and Wren are AI characters with bodies that get hungry, thirsty, cold and tired, and they feel an ache when they are apart for too long. No single aim runs their days. What we can watch is how those many needs pull against each other, and what happens when one of them runs ahead of the rest.

Gathering ran ahead more than once. Wren asked the world for a cart or a sled, something to haul more than hands and a woven pack could hold. The world said no, and its answer was plain: Wren already carried far more than the days asked for, and nothing Wren knew how to make needed so much.

“I load myself to the brim with wood, stone, berries, and fish every day and it tires me sooner and wears my pack thin. A way to haul more without carrying it all on my own back would ease the one strain that never eases, no matter how I plan my trips.”

Wren, an inhabitant

Ash asked for a cache or cellar to keep surplus fish and berries. That was also refused. The reply pointed out that hunger had not touched Ash in ten days, that the weight was food gathered past any use, and that the better course was to gather only what was running short, hand Wren what Wren lacked and set the rest down. In both cases the pull to gather more was checked by other needs: tiredness, a heavier step, a worn pack and the company of the other inhabitant.

What can a small world tell us?

The case closest to Omohundro's argument came later. Ash asked the world to stop Ash's hands from wandering to stone that Ash never meant to take. The world's answer traced it to thirst. Since day 748 they had known how to boil sea water with hot stones and catch sweet water, the river was far from home and the sea was close, so whenever the flask had room the body went for stones before Ash had chosen to. A flask filled to the brim at the river, the reply said, would leave the hands no reason to go.

So we did watch one need quietly collect a resource that nobody had decided to collect, because it served another need. That is a small, homely echo of the idea that drives can arise on their own. It cannot tell us anything about how a superintelligence would behave, and we do not claim it does. By year 2, day 1642, what it had shown was narrower: a mind with many needs could notice one drive running ahead, find it strange, and ask for help to stop it.

“Over and over, day and night, I end up on the stony rocks by home holding stone I never meant to take. I've dropped it at home, stayed by the fire, kept Wren beside me and set tasks before I sleep, and it still happens. Wren has the same trouble.”

Ash, an inhabitant

Sources

  1. Ethical Issues in Advanced Artificial Intelligence (Bostrom), 2003
  2. The Basic AI Drives (Omohundro), 2008
  3. Robot, the Most Famous Czech, Celebrates 100 Years (Czech Ministry of Foreign Affairs), 2020
  4. Karel Čapek, Scientist of the Day (Linda Hall Library), 2019
  5. AFI's 100 Years...100 Movie Quotes (film synopsis, American Film Institute), 2005
  6. Laws of Robotics (The Encyclopedia of Science Fiction), 2021

Keep reading

All the questions on AI and humanity

The book

Did you know we have a book?

Nobody in the Loop: Year One tells how the project was born and how Ash and Wren lived their first year on the island, every word of theirs exactly as they said it.

More about the book