GCHQ

Code-breaking is sometimes depicted as highly technical – more ‘Billion Dollar Brain’ than James Bond – and therefore perhaps a little dull. But much of the GCHQ story involves dramatic incidents experienced by individual sigint operators in forward locations, including in submarines and aircraft. However it was done, gathering sigint almost always involved a three-stage process. First, someone had to listen in to and record the intercepted message. Throughout the Cold War this person was often the Godforsaken GCHQ ‘operator’ who sat for eight hours at a time in front of a rack radio made by Racal. With headphones on and the volume turned up to ‘max’ he or she endured the freezing cold of the German winter and the unbearable heat of the Iraqi summer. Once the message was captured it was passed back to Cheltenham for processing. If it was in code, it might be given to X Division, a section staffed by ‘boffins’ with vast computers whose power far outstripped that available to ordinary scientists. Finally, intelligence analysts would try to compose the resulting material into useful summaries. Stamped with an excruciatingly high security classification, it was then circulated to Cabinet Ministers, defence chiefs and senior policy-makers. Often, only a few hours after they had been read by the ‘high-ups’, the summaries were whisked away in ‘burn-bags’ and consigned to vast incinerators to protect their secrecy.

GCHQ is also synonymous with the mysterious international network known as ‘Echelon’, run by British and American intelligence. Echelon is the world’s largest information ‘vacuum cleaner’, drawing in huge amounts of communications – an estimated five billion intercepts every day. Yet much of what we have come to believe about this network is wrong.15 The Anglo–American sigint relationship is often portrayed as a cosy affair of affable, pipe-smoking professor types. In fact, the politics of intelligence was often opportunistic and harsh. Secretly, the British and Americans worked together to read the traffic of their own minor allies, including France and West Germany. Even at the top, relations between the two main partners, Britain and the United States, could turn nasty and involved sharp disagreements.

What bound Britain and America together in the world of signals intelligence was realism, not romanticism. Anglo–American intelligence cooperation was about trading ‘terrain for technology’. America had its own vast code-breaking organisation, the National Security Agency (NSA), with infinitely more resources than the British. However, the American code-breakers needed remote outposts in Britain’s ‘residual empire’ at which to base their listening stations, and they rewarded GCHQ handsomely with access to remarkable technology. Some locations, such as Cyprus, were so important to the collection of sigint that UKUSA actually helped to shape the international politics of the region. In 1974, faced with a financial crisis, the British government formally decided to withdraw from its bases in Cyprus in order to save money. Within days, Washington told London that this decision was not acceptable and they must stay. The reason was simple. The sigint bases that allowed America to listen in to the Middle East were quite indispensable. In 2009, more than thirty years after the British government’s decision to withdraw from Cyprus, the sigint bases are still there, and have grown considerably in size.

Cold War espionage activity enjoyed a high profile. British defectors such as Guy Burgess and Donald Maclean hit the headlines in the 1950s. The 1960s opened with the shooting down of the American U-2 spy plane piloted by Gary Powers, the CIA’s fiasco at the Bay of Pigs and the Profumo affair. Yet GCHQ managed to avoid the glare of unwelcome publicity until the last decade of the Cold War. Its journey from the shadows into the spotlight only began in 1976, when the radical journalist Duncan Campbell revealed its intelligence operations on Cyprus in an article in Time Out magazine. This led to the infamous ‘ABC trial’, at which Campbell and his associates were prosecuted under the Official Secrets Act. Thereafter, GCHQ’s hopes to return to obscurity were dashed by the Geoffrey Prime affair in 1982. Prime, who revealed the innermost working of America’s latest multi-billion-dollar sigint satellite programme to the Soviets, was one of the most damaging moles ever recruited from inside British intelligence. Just as the Prime case subsided, any hopes of a return to anonymity were obliterated by Margaret Thatcher’s controversial decision to ban trade unions at GCHQ.

Expensive technical agencies such as GCHQ and America’s NSA were obvious targets for cuts at the end of the Cold War. At the same time, both agencies were struggling to cope with the pace of the global information-technology revolution, that had made access to high-grade encryption easy for the private individual. All this, together with the exponential growth in internet traffic, threatened to make the work of GCHQ and NSA impossibly difficult. Soon the world was sending several million emails a second, and not even the great sigint leviathans could read them all. The days of the super-secret sigint agencies seemed numbered. However, in the 1990s Britain’s prominent role in the wars in Bosnia and then Kosovo reminded government that the need for sigint is perennial. In these Byzantine conflicts, the radio experts at Cheltenham were never quite sure which of the many different former Yugoslavian factions their various friends and allies were supporting.

Bitter conflicts such as Bosnia helped to convince Whitehall and Westminster that GCHQ was worth new investment. In 1996, under the direction of Sir David Omand, GCHQ began to develop plans for a remarkable new intelligence headquarters that quickly became known as ‘the Doughnut’ owing to its circular design. The intention was to bring all the staff together under one roof for the first time. Absorbing no less than fifteen miles of carpet and several hundred miles of fibre-optic cabling, ‘the Doughnut’ constituted the largest secret intelligence headquarters outside the United States. However, by the time it was completed in 2003, it was already too small. GCHQ had by then undergone a crash expansion following the 9/11 terrorist attacks. Its employees, now numbering more than 5,200, were soon ‘hot-desking’. A shanty town of subsidiary buildings is already springing up around the new headquarters.

Today, in somewhat cramped circumstances, GCHQ struggles with some of the most difficult issues of the twenty-first century. Not only is it the leading edge of Britain’s struggle against al Qaeda, it is also involved in fundamental issues of freedom and privacy that will shape the future of our society. Over the last decade, Britain has engaged with global e-commerce and finance more enthusiastically than perhaps any other country in the world. Our porous electronic borders present their own enormous problems. Globalisation, and in particular the global communications revolution, has brought many benefits, but it has also allowed miscreants to communicate and organise anonymously. The need for GCHQ to monitor both terrorists and organised crime means that the distinction between domestic and foreign communications has less meaning than it once had. GCHQ used to be a wholly outward-looking foreign intelligence service, but this is no longer the case.

Who will rule the internet? Will ordinary citizens be allowed genuinely confidential communication? Would ID cards erode our privacy or extend our security? These are some of the questions that GCHQ ponders daily at the beginning of the second decade of the twenty-first century. Britain is already one of the most watched societies in the world, and some would argue that it is now addicted to surveillance. In 2008, Britain announced a £12 billion project to modernise the interception of telephone calls and email. The following year GCHQ announced a remarkable project entitled ‘Mastering the Internet’ that collects the details of Britain’s communications and internet traffic for security purposes. Even Britain’s Director of Public Prosecutions thought things had gone too far. Tasked with taking the lead on technological aspects of intelligence, GCHQ now finds itself at the centre of controversies that are of immense public importance. Accordingly, the time is ripe to trace GCHQ’s long and secretive journey from the nissen huts of Bletchley Park – via the Cold War – towards what now looks increasingly like a Brave New World.

THE 1940s BLETCHLEY PARK AND BEYOND

1 Schooldays

‘How wonderful!’ I said. ‘Do you mean we’re overhearing Portsmouth ships trying to talk to each other – that we’re eavesdropping across half South England?’

‘‘Just that.’

Rudyard Kipling, ‘Wireless’, 19041

In December 1902, Guglielmo Marconi made history by sending the first wireless radio message across the Atlantic. Remarkably, only two years later, Rudyard Kipling foretold the possibility of exploiting such radio messages to gather intelligence. In 1904 he published a short story entitled ‘Wireless’ that focused on intercepting communications sent from Morse equipment on board Royal Navy ships off the Isle of Wight. Kipling is thought of as a quintessentially late-Victorian author, but here he looks to the future, more in the manner of H.G. Wells, as his characters fret over technical matters such as induction and radio frequencies. To the readers of this fictional first instance of radio interception, the process seemed utterly magical. The Morse instrument ‘ticked furiously’, and one of the listening party observes that it reminds him of a séance, with ‘odds and ends of messages coming out of nowhere’. His companion retorts that spiritualists and mediums ‘are all impostors’, whereas these naval messages that they are eavesdropping on are the real thing.2

Kipling’s ‘Wireless’ is the first public discussion of the secret business of signals intelligence, or ‘sigint’. The magical process of extracting information from the ether would be one of the twentieth century’s most closely guarded secrets. Initially, producing ‘sigint’ only required equipment that would allow a third party to eavesdrop on a conversation broadcast by a radio transmitter using ‘wireless telegraphy’, but as this possibility became more widely known, communicators often resorted to using cyphers to keep their messages private. Thereafter, producing sigint usually required skilled listeners to capture the message and then a team of code-breakers to unscramble it. If the message was sent by cable rather than wireless, the listening-in process could be no less difficult than the code-breaking, or ‘decyphering’.

What did Britain’s code-breakers make of Kipling’s public airing of their black arts? The simple answer is that there were none to ask. Indeed, there had been no British code-breakers for more than fifty years. In the distant past, Britain had possessed a ‘black chamber’ in which skilled ‘cryptanalysts’ had broken the codes contained in diplomatic correspondence and private letters. These arcane skills resided in the ‘Secret Department’ of the Post Office. However, in 1847 this was exposed in a scandalous episode when the House of Commons heard that the Home Secretary had ordered the interception of the private correspondence of the heroic Italian nationalist in exile, Giuseppe Mazzini. Shocked Members of Parliament ordered an inquiry, leading to the closure of the ‘Secret Department’, just as the telegraph initiated what we now understand as a Victorian communications revolution. By 1904, Britain had been without a code-breaking centre for more than half a century3

The immediate origins of MI5 and its sister service SIS (often known as MI6) can be traced to scares about German espionage in 1909. But British code-breaking was not revived until the very eve of the First World War. On 2 August 1914 the British Army set up a secret code-breaking section called MIlb. Soon, specialist Army units at various locations in Europe and the Middle East were busy intercepting German radio communications. One of the largest sites was the intercept station in Mesopotamia. In December 1916 the military code-breakers of MIlb were given a fabulous Christmas present when the drunken chief of the German signals organisation in the Middle East sent all his Radio Operators a seasonal greeting using the same obvious formula in no fewer than six different codes. Up until that point the British had only been able to read one of these codes, but with these clues they could read all six. In the First World War, the Second World War and again in the Cold War, poor discipline by the human operators often proved to be the great weakness in otherwise impregnable cypher systems.4

The Royal Navy code-breakers, who had established themselves in the Admiralty’s ‘Room 40’, achieved even greater success. Famously, they broke the ‘Zimmermann Telegram’, a message sent from the German Foreign Minister, Arthur Zimmermann, suggesting an alliance between Germany and Mexico against the United States. As an inducement, Mexico was to be offered the return of her lost territories in Texas, New Mexico and Arizona. These revelations, made public in March 1917, were central in bringing the United States into the First World War on the side of Britain and France. The American entry into the war, together with a tightening blockade, persuaded Germany to seek an armistice the following year. The code-breakers of Room 40 celebrated with champagne. There are few more significant examples of the direct impact of code-breaking upon international relations.5

In 1919 the British government’s Secret Service Committee, chaired by Lord Curzon, the rather formidable Foreign Secretary, recommended that a unified peacetime code-breaking agency should be created. This involved the difficult merger of two quite separate organisations. The head of the Army code-breakers, Major Malcolm Hay, was awkward and argumentative, while his naval equivalent, Commander Alastair Denniston, proved to be suave and diplomatic. Denniston secured the job as chief of a new combined code-breaking organisation, which initially consisted of around two dozen intelligence officers and a similar number of clerical staff, and found himself installed in splendid accommodation at Watergate House in The Strand, next to the Savoy Hotel. Formed on 1 November 1919, the new organisation was given the name ‘Government Code and Cypher School’, or GC&CS, which was not inappropriate, since the leading code-breakers devoted a great deal of time to the patient training of new initiates.6 Both during the First World War and in the interwar period about half the staff of GC&CS and its predecessors were women, mostly in the clerical grades.

Almost immediately, GC&CS adopted a disingenuous description of its duties that would remain in place until the 1980s. Publicly, its functions were described as merely defensive; in other words, it was to assist in the provision and protection of codes and cyphers used by government departments. However, its more secret duty was to give priority to offensive activity, namely attacking the cypher communications used by foreign powers. GC&CS gradually shifted its focus to diplomatic traffic, and at the suggestion of Lord Curzon it was transferred to the control of the Foreign Office. It seemed natural that within the Foreign Office structure it should be placed under the supervision of Britain’s traditional overseas intelligence service, SIS, which recruited human spies. But a subliminal naval influence remained. The talented Chief of SIS, Mansfield Cumming (known within the organisation as ‘C’, the name by which the head of SIS would continue to be called), was a former naval officer. Cumming died in harness in 1923 and was succeeded by another sailor, the former head of Naval Intelligence, Hugh ‘Quex’ Sinclair. Naval intelligence and naval signals officers continued to exercise a profound influence on GC&CS and its successors as late as the 1970s.

The means by which Britain collected its intelligence was changing. During the First World War, much of its intelligence work had involved overhearing military wireless messages by means of receiving stations scattered around Europe, the Mediterranean and the Middle East. The shift to diplomatic traffic meant undertaking more work on encyphered diplomatic telegrams sent by cable. Each country had teams of code clerks who carefully encyphered diplomatic messages before they were sent by telegram using a worldwide network of cables. Although government cable censorship had officially ended in 1918, a private arrangement meant that all the commercial cable companies secretly handed over their traffic to GC&CS for copying. Most of the foreign embassies in London used cable companies to send their encyphered messages, and British dominance of international telecommunications networks meant that many of the world’s messages travelled over British cables at some point. Private companies such as Standard Cable & Wireless Ltd were almost an integral part of the worldwide British sigint system. This secret state-private network remained hidden until it was exposed by the journalist Chapman Pincher in February 1967 in the Daily Mail under the headline ‘Cable Vetting Sensation’.7

In 1925 both SIS and GC&CS were moved into Sinclair’s new secret service headquarters at Broadway Buildings, opposite St James’s Park tube station, which its occupants thought ‘more dingy than sinister’. The walls of the corridors were painted dark brown to a height of about four feet from the floor, and the ancient lifts moved between the many storeys with a slow clatter. The code-breakers were given the third floor. From here, the sigint product, which consisted of the verbatim text (or sometimes summaries) of the messages of foreign governments was distributed around Whitehall in files with special blue jackets that became known as ‘BJs’. GC&CS worked on the cyphers of many countries in the interwar period, including those of France, the United States and Japan, since they all shed light on international affairs; but the most important were those of Russia.8

Both MI5 and SIS, together with intelligence officers from the three armed services, were obsessed with the threat from Bolshevik Russia in the interwar period. GC&CS followed suit. There were good reasons for making Moscow the pre-eminent target. Bolshevik agents were actively seeking to subvert the British Empire, and sigint produced operational intelligence that could be used to thwart these plots. Alastair Denniston enjoyed a major advantage, having recruited Ernst Fetterlein, the Tsar’s leading code-breaker, when he fled Russia after the Revolution of 1917, and in the 1920s GC&CS was successfully reading Soviet diplomatic cyphers. Several times during that decade the British government directly accused the Soviets of underhand activities in London, making use of these intercepts and referring to them openly. In 1923, for example, Lord Curzon publicly quoted Soviet messages intercepted by GC&CS stations in India. The Soviets responded by changing their cyphers, but Fetterlein simply broke them again.9

However, in May 1927, a year after the General Strike, a disastrous row erupted over secret support from Moscow for the strikers and the distribution of subversive propaganda in Britain. A veritable centre for Soviet subversion was being run under the cover of its Trade Mission, located in the Arcos building in Moorgate. The building was raided on 12 May, but advance warning allowed the Soviets to destroy most of the incriminating material. The Prime Minister, Stanley Baldwin, was embarrassed. He ardently desired to break off relations with Moscow, but having failed to garner any incriminating evidence from the Arcos raid, he turned to the priceless intercepts provided by GC&CS. To the dismay of the code-breakers, Baldwin and his Foreign Secretary, Neville Chamberlain, read out four decyphered Soviet telegrams in Parliament in order to make their case. Alastair Denniston was especially bitter about this flagrant compromise of GC&CS secrets.10

Henceforth, the Soviets changed their cyphers and deployed more secure systems for communications with diplomatic and commercial missions overseas, including their intelligence stations. They now used the ‘one-time pad’ for their more important communications. The one-time pad was a breakthrough system created by an American army officer, Major Joseph Mauborgne, during the First World War and widely adopted by other powers. It involved using a sheet of random numbers to encypher a message. Each letter in the message was given a number. Each number was then added to another from a stream of random numbers taken from a sheet on the one-time pad. The result was a sheet of text that consisted simply of groups of five numbers, one after another. Recipients could decode the message if they possessed the same sheet from the same one-time pad. If that sheet was used only once – hence the name – and for a single message, the lack of repetition prevented decryption. In short, the code was unbreakable. The disadvantage was that it was slow and cumbersome, and therefore it was reserved for high-grade secrets. Moreover, vast numbers of pads with lists of random numbers were required. No country, not even the security-obsessed Soviet Union, could send all its communications by this means.11

Nevertheless, after 1927, few Soviet diplomatic messages were being read by GC&CS. The only high-grade Soviet traffic that was decyphered were the messages of the Comintern, the part of the Soviet Communist Party that dealt with relations with Communist parties overseas. This effort was led by John Tiltman, a brilliant major from the Indian Army who had been running a small but successful interception effort in north India during the 1920s. In 1929 he was brought back to London to lead an expanded operation against Comintern communications (which were code-named ‘Mask’). This allowed the British government to learn of the secret subsidies paid by Moscow to the Communist Party of Great Britain and its newspaper, the Daily Worker. It also contributed to important successes against major Comintern agents in imperial outposts and international centres such as Singapore and Shanghai.12

Faced with the real threat of active subversion throughout the British Empire by the Comintern, GC&CS paid limited attention to military matters or the rise of the Axis until the mid-1930s. Germany, Italy and Japan were a remarkably low priority. Admittedly, a small naval section of GC&CS had been set up in 1925, and its most important work was done overseas by naval officers like Eric Nave, based in Hong Kong. From here they had ample practice at following military operations, because of the extensive fighting in Manchuria during the 1930s. Italy’s attack on Abyssinia in 1936 provided a new target for British code-breakers in the Middle East, located at sites such as Habbaniya in Iraq and Sarafand in Palestine. Remarkably, and despite the growing importance of air power, GC&CS only developed an RAF section in 1936, under Josh Cooper, a young and talented code-breaker who had joined the organisation a decade earlier with a First in Russian from King’s College London.13

Cyphers were important to the Axis military powers. One-time pads were slow and cumbersome. Moreover, they were out of step with the emerging new methods of warfare. Blitzkrieg, for example, required armoured forces to move forward at lightning speed, coordinating their activities with artillery and air support. So the pressure was on to find a way of making the growing volume of military radio traffic unintelligible to the enemy. Most developed countries turned to cypher machines to make their immense volumes of traffic secure.14 Complex cypher machines had been pioneered by banks and businesses – banks had long used fairly simple cyphers to keep commercial matters secret. In the 1920s, the German military adapted a Dutch invention to produce the Enigma cypher machine as an alternative to laborious hand cyphers. In fact, the first Enigma machines were sold commercially, and were widely used by banks and businesses. Enigma was what we now recognise as a ‘commercial off-the-shelf solution’ to a difficult military problem.15

The Enigma machine itself looked like an early typewriter in a square wooden box, but with a keyboard set out in alphabetical order rather than the traditional ‘QWERTY’ arrangement. As each letter key was depressed a set of lights that corresponded to the alphabet lit up, seemingly at random. The innovation was the rotors, which looked like fat metal wheels, embedded in the top of the machine. These rotated and scrambled the message in a highly unpredictable way. There were initially three – later four – rotors, with twenty-six positions relating to the letters of the alphabet. These moved round in a stepping motion that generated a cypher with an enormous number of possibilities. Moreover the complex nature of the rotation caused subtle changes in the stream of material, creating substantial headaches for any would-be code-breaker. The Germans were not alone in developing cypher machines. The British and Americans developed similar devices, respectively called the Typex and Sigaba.16