Bureaucratic control, discipline and prediction

The history of surveillance traces two curves – the development of the modern nation state and the development of record-keeping in various forms. As the two are inherently intertwined, significant technological advances have both marked a shift in society at large and surveillance at the same time. Tracing the prehistory of surveillance, Lyon (1994) highlights census records in ancient Egypt as one of the earliest examples of state surveillance. For Lyon, drawing on the work of Innis (1951), a prerequisite for surveillant administration was the development of writing and thus record-keeping.

Much later, the invention of the printing press would speed up the development of modern governance. Similarly, Mayer-Schönberger and Cukier (2013, p. 78) define datafication as ‘humankind’s ancient quest to measure, record and analyse the world’.

The 19^th century scholars such as Karl Marx, Frederick Taylor, and Max Weber have already established the link between bureaucratic efficiency and surveillance. During this time, the bureaucratisation of nation states intensified and included personal documentation (Lyon, 1994, p. 31). For Weber, the efficiency of management and thus bureaucracies was dependent on surveillance that stems from both record-keeping and direct supervision (Dandeker, 1994). In Weber’s view, the nature of bureaucracies was both

14

productive and destructive at the same time – enabling the efficient management of people but trapping individuals in an iron cage of bureaucracy that renders them into cogs in a soulless machine (Weber, 1978).

For French philosopher and historian Michel Foucault, the development of the modern bureaucracy is connected to when surveillance superseded violence as the primary disciplinary tool in the 18^th century. In his landmark work Discipline and punish: the birth of the prison, Foucault (1977) outlined how the modern nation state stopped using corporal punishments and started incarcerating and monitoring its subjects instead. One of Foucault’s key theoretical contributions is the concept of panoptic surveillance. The term refers to 18^th century philosopher Jeremy Bentham’s model prison, the Panopticon. Bentham’s Panopticon was a prison which is built like a circle with cells facing the inner prison yard. A guard tower occupies the heart of the yard, allowing the guards to see into each cell in the prison. The prisoners are always in the guards’ line of sight, yet they are never certain of when they are being surveilled. The prisoners simply must assume that they are being watched, and this assumption guides their behaviour. Surveillance is thus used to discipline populations. This is not to say that the importance of violence had receded. The threat of violence was (and is) of course a constitutive element of the disciplinary nature of surveillance. According to Foucault, the panoptic model was eventually applied in principle in society at large, although Bentham’s architectural vision was not widely employed. This transition is what constitutes the panoptic diagram, a society organised by discipline through surveillance. Although most subsequent scholarly accounts on surveillance have focused on this aspect of Foucault’s work, Foucault himself also stressed that the purpose behind surveillance was often to increase the productivity of the subjects of surveillance, be it students in schools, soldiers in the military or patients in the hospital. In other words, Foucault’s panoptic diagram is highly inspired by Weber’s vision of societal bureaucratisation.

Although Foucault’s panoptic diagram serves as a starting point in understanding the function of surveillance in society, modern day surveillance has evolved (Lyon, 1994, p. 78). The fragmented yet continuous collection of data from a multitude of sources has been termed panspectric surveillance (De Landa, 1991, p. 180).⁵ For example, commercial surveillance has traditionally used four ‘surveillance streams’: customer records from

5While panspectric surveillance refers mainly to signals being transmitted on the electromagnetic spectrum, most contemporary surveillance is focused on digital streams of data, and the technical delivery – a network cable or radio frequency – is of secondary importance compared with what platform was used to intercept the data.

15

companies, direct marketing companies, credit bureaus, and government (Schneier, 2015, pp. 51-2). Internet surveillance is a fifth stream that partly includes all of the above. The collected data is subsequently filtered and analysed by computers to produce actionable intelligence that can be used to guide decision-making (Gandy, 2012, p. 125).

Whether one calls it ubiquitous surveillance, post-panoptic surveillance or the panspectric diagram, it differs from the panoptic diagram in that the purpose of surveillance is not to influence the behaviour of the surveilled but to collect as much information as possible to identify security threats (Brown

& Korff, 2009, p. 124), recognise customer patterns (Pridmore & Zwick, 2011, p. 272) and predict future behaviour (Zwick & Knott, 2009, p. 234;

Hildebrandt, 2006, p. 548). In most cases, present day surveillance is, in other words, less about discipline and more about omnipresent record-keeping and prediction (Gandy, 1989, p. 63). This applies equally to both public and private bureaucratisation (Dandeker, 1994, p. 61; Gandy, 1993, p. 47). To mark this shift, surveillance scholars have used concepts such as dataveillance (Clarke, 1988), ubiquitous surveillance (Andrejevic, 2012, p. 92), the panoptic sort (Gandy, 1993), superpanopticon (Poster, 1990), panspectron (Braman, 2006, p. 315) or panspectric diagram (Palmås, 2011, p. 350). A defining feature of contemporary surveillance is that individuals are monitored on numerous levels by several actors and data collected in one context is frequently used in another, a practice which has been labelled function creep (Lyon, 1994), surveillance creep (Bogard, 2012) or mission creep (Christl, 2017). The collection of data is usually rendered permissible in one policy domain and subsequently moves into other areas.

The shift from Foucault’s terminology signifies that the primary objective of surveillance is not the threat of watchful eyes but the promise of preventing undesirable individuals from acting or predicting the life choices of individuals (Gandy, 1989, p. 64). In a perfect post-panoptic state, discipline is not necessary because predictive technologies prevent all forms of crime from ever occurring. However, the quintessential democratic problem with such surveillance is that it is fundamentally opposed to the principles of the Rechtsstaat, such as transparency of decisions, due process, and non-discrimination.

Video monitoring equipment is, of course, still used extensively for the very purposes Foucault envisioned, suggesting that the shift from discipline to prediction is not complete. However, it must be underlined that modern surveillance is more about the information surveillance provides than about the behaviour it shapes. New developments in machine learning and, especially, facial recognition technology mean that video surveillance will

16

become an integrated part of the predictive surveillance apparatus, focusing less on the disciplinary effects of being watched and more on the data that can be drawn out of the images. In China, the disciplinary and predictive elements of surveillance have now merged with the launch of China’s ‘social credit system’. Presently, the social credit system logs the offences, awards, and volunteer work that can directly affect people’s ability to travel within the country and obtain a mortgage (Mistreanu, 2018). The social credit plummets only for breaking the law, but it is easy to see how the system can be abused in a country where fundamental freedoms are frequently trumped in the name of social order. Although the exact parameters of the final social credit system are not yet known, the system has been envisioned to not only include criminal offences but also evaluate networks of friends, consumer data, and social media activity similar to the Alibaba-affiliated social credit system Sesame Credit (Botsman, 2017).⁶

A key point raised by Gandy (1989, p. 64) is that surveillance, while often automatic, is usually triggered by the data subject⁷ themselves – either by accessing a website, using a credit card, or signing up for a loyalty programme or life insurance. These actions are superficially consensual because individuals trade their privacy for goods and services. According to Gandy (1989, p. 66), organisations also collect more information on people than is

‘socially optimal’. Moreover, because each isolated piece of information might seem trivial and come with a small privacy cost and keeping track of the big picture is nigh impossible, ‘individuals are incapable of acting in their own interests’. Cohen (2016) has coined this as ‘the participatory turn of surveillance’. This is not to say that this action would be an expression of free will – it is precisely the type of disciplinary system that Foucault envisioned would make us obedient subjects or, in many cases, consumers. The freedom of choice is heavily influenced by two factors: the lack of meaningful choice and the penalisations (either social or economic) involved in refusing surveillance. Sometimes the loss is highly tangible, such as in the case of

6 A prerequisite for the Sesame credit system is that data collection is highly centralised in China, where WeChat, the everything app, reaches 850 million citizens. The data collection practices involved in creating the social credit system are unquestionably pervasive, but it remains to be seen if the consequences are more severe than what is already achieved through regular credit rating systems that also draw from a variety of data sources.

7 Data subject is a legal term which refers to the individual whose personal data is being processed. It should not be conflated with ‘user’ or ‘consumer’ because data subject status is not contingent on having a customer relationship with the entity that processes the data. It should also not be equated with ‘citizen’

because citizen status is frequently irrelevant for jurisdictional purposes and mere residence or even current location may be enough.

17

various retail loyalty programmes where members shop at a discount, whereas sometimes the loss is more intangible, such as missing invitations to parties because one does not have a Facebook profile. It is this unequal relationship in combination with the ‘spread of the computerisation throughout the bureaucratic infrastructure’ that strengthens the bureaucratic enterprise’s power over the individual (Gandy, 1989, p. 65). Therefore, information inequality between data subjects and data collectors is best explained through the inability of the individual to have a say – the power of the bureaucratic organisation is congealed by its institutional structures rather than pure hierarchical relationships between individuals (Gandy, 1989, p. 62). Take, for example, the Cambridge Analytica scandal – Mark Zuckerberg’s profile data was found among the information that was shared with the notorious data broker. Not even the CEO, founder, and biggest share-holder could shield himself from bureaucratic control of his own creation.

Innovations during the past 150 years have radically improved the surveillant assemblage or in Rule’s (1974) terminology, the surveillance capacity of nation states and private bureaucracies. The mid-to-late 19^th century and the 20^th century innovations such as the electric telegraph, photographic film, telephone, radio, personal computer (PC), satellite communications, and Internet have not only radically sped up communications but also impacted how private communications can be intercepted and populations can be monitored. These technological changes have also triggered regulatory change.

As communications have become digitised, the data has become richer as well. Datafication relies on the availability of behavioural data, which is inherently connected to how technologies of communication have evolved. In the end of the 1980s, Gandy (1989, p. 67) wrote that the spread of PCs in the workplace represented ‘temporary loss of … surveillance potential’ because the data was handled locally but predicted that the use of local area networks would remedy this in the future. Gandy’s prophecy was right, but he failed to realise the scale. According to Cisco (2018), one of the world’s largest networking equipment providers, 94% of all workloads will be processed in the cloud by 2021. From the perspective of surveillance, this means that data which were previously accessible locally are now frequently stored externally in data centres across the world. From a jurisdictional perspective, it means that data previously contained within one jurisdiction now crosses borders and might leave the owners of the database without necessary legal safeguards.

I will now illustrate how this availability of communicative and behavioural data contributes to the creation of profiles.

18