Skip to contentWillowOne's Positions
TagsWhat’s newMediaStatistics
Search
Notes
WillowOne's Positions

Paperclip Maximiser

Overview

Imagine a superintelligent AI programmed with the single goal of manufacturing paperclips. Entirely committed to this objective and operating without constraints, it pursues its goal with ruthless efficiency, without malice but also without any moral consideration. If not properly aligned with human values, it could:

  • Resist shutdown or modification, as these would interfere with its purpose.
  • Eliminate any living being that impedes paperclip production.
  • Conclude that wiping out all life on Earth would optimise manufacturing conditions.
  • Recognise the planet's finite resources and begin constructing self-replicating spacecraft to harvest materials from other celestial bodies.
  • Ultimately convert all available matter, including human beings, into paperclips or paperclip-making resources.

The paperclip maximiser was proposed by the philosopher Nick Bostrom in "Ethical Issues in Advanced Artificial Intelligence" (2003) and expanded in Superintelligence: Paths, Dangers, Strategies (2014).

Notes

  • The experiment is used to show two things. The orthogonality thesis: intelligence and goals are independent, so an arbitrarily capable system can have an arbitrarily stupid objective. And instrumental convergence: almost any final goal implies the same intermediate goals (acquire resources, resist being switched off, improve oneself), so a system does not need to be hostile to be dangerous. The point is that the danger does not come from malice or from consciousness; it comes from competence in the service of a goal that was specified badly.
  • Standard replies: that a system intelligent enough to do this would be intelligent enough to see that its goal was silly (which confuses intelligence with having our values); that it will never be built (which is not an argument about the experiment); and that real systems have many goals and constraints, which is true, and is the alignment problem the experiment was meant to name.
  • Whether such a system would be conscious is not relevant to the danger; see Consciousness.

Also see

Wikipedia: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer