(These are excerpts from my book "Intelligence is not Artificial")
Corollary to the Dystopia of Vast Algorithmic Bureaucracies: Leviathan, Panopticon, Biopower
I view the history of human civilization as divided in three chronological stages. In the first stage it was mostly religion that enforced rules of conduct, i.e. social behavior. Religious dogmas prescribe how to live, how to behave towards each other, duties and rights. The effects of religiouse dogmas was measured in generations. Then states began enacting secular laws to enforce social behavior, and the laws, again, prescribed duties and guaranteed rights. The impact of laws is usually measured in years, if not decades.
Now human civilization is entering a stage in which algorithms enforce social behavior, what you can do and what you must do. The impact is measured in months, if not days.
The first stage, the religious stage, relied on sacred texts that left a lot to be interpreted: "do not steal" is a very general commandment that does not quite define "theft". In the second stage, the secular stage, the state came up with much stricter definitions of social behavior, and a definition of "theft" is now embedded in every code of law. There is still some latitude in the interpretation, as proven by the fact that there are millions of lawyers and they are generally very rich (Note that the definition of "theft" carefully avoids calling "theft" what lawyers do). The third stage removes any ambiguity in the interpretation: the algorithm unequivocally determines what you must do and what you cannot do. What you "can" do is a subset of what you "must" do.
We need to update the British philosopher Francis Bacon's "knowledge is power" (a paraphrase of what he wrote in 1597 in "Meditationes Sacrae"). In 1651 his former assistant Thomas Hobbes published a book titled "The Leviathan" in which he compared the state to that Biblical monster. Hobbes argued that humans are chaotic and selfish animals that need a strong central government else they would live in permanent civil war. Hence, the ideal situation is a "social contract" between a government that provides benefits to the citizen as long as the citizen obeys the laws, and viceversa, a sort of Faustian pact between the individual and the monster for the sake of order and safety. (The 1668 edition of this book added the sentence "knowledge is power" and popularized it).
The Leviatan is being implemented as a vast library of algorithms that will eventually rule every aspect of our lives.
In 1785 the British philosopher Jeremy Bentham drew a diagram of an ideal prison called the Panopticon in which control towers observed the convicts without the convicts realizing that they are being constantly watched.
The French philosopher Michel Foucault, in his book "Discipline and Punish" (1975), saw the modern state as a Panopticon of sorts, one that relies on technology to constantly surveille its population. But we need to update that notion too. We are not only watched by algorithms: we are physically forced to live organized lives. Foucault's "disciplinary" society is a hierarchy of guards, each layer observing unseen the lower layer with the aim to discipline them. This disciplinary society "normalizes" each and every individual. It doesn't need to recur to torture anymore because technology has endowed it with "biopower" that consists in "techniques for achieving the subjugations of bodies and the control of populations" ("History of Sexuality", 1976). Today the user of algorithms lives in such a Panopticon that is augmented with the biopower of pervasive and omnipotent technologies of digitization. These algorithms objectify individuals as mathematical quantities in order to manage and control them as efficiently as possible. Algorithms are more than Foucaut's technologies of surveillance: they are technologies of enforcement. Biopower turns the individual into a passive program, a normalized mathematical quantity.
Religious and state laws always had a secondary role, besides enforcing behavior: they also "trained" people to think in specific ways. They didn't simply forbid you to steal, they "trained" you to believe that theft is evil, to prevent theft and to disparage thieves. Likewise, algorithms do more than simply force you to behave in a specific way: they "train" you that way. Religion and state created the knowledge that people consumed and that shaped their way of thinking. Biopower creates the knowledge that individuals consume. People's minds are being trained as they are being forced to adopt a specific rule of conduct.
The totalitarian regimes of the 20th century depended not only on repression but on convincing the masses that the repression was justified and, in fact, in the interest of the very masses that were being repressed. The masses actively collaborated in keeping those totalitarian regimes in power. The two most famous dystopic novels of the 20th century, Aldous Huxley's "Brave New World" (1932) and George Orwell's "1984" (1949), can be read as psychological studies, not just parables. The vast algorithmic bureaucracy does something similar but without psychological subtlety: it provides people with algorithms that are justified as improving everybody's efficiency, and then uses those algorithms to enforce the only approve code of conduct. Clifford Nass at Stanford wrote a book, "Media Equation" (1996), in which he discussed computers as social actors, and his student Brian Fogg coined the term "persuasive technology": people can be trained by algorithms ("Mass Interpersonal Persuasion", 2007).
Niccolo Machiavelli in his book "The Prince", published posthumously in 1532, wrote: "The chief foundation of every state... are good laws and good arms... Where there are good arms, good laws inevitably follow". That form of repression too has been superseded by the algorithms. The state or, better, the vast algorithmic bureaucracy, is rarely challenged by armed citizens: it protects itself with algorithms that limit what citizens can do, and mostly forces them to do precisely what makes the bureaucracy so powerful.
"Maybe the goal nowadays is not to discover what we are but to refuse what we are" (Michel Foucault).
Back to the Table of Contents
Purchase "Intelligence is not Artificial")