GCHQ

Where would GCHQ’s new centre be? What it craved was a site in central London, next to the policy-makers, but even with the post-war demobilisation of many government departments, nothing suitable could be found. The solution was what John Betjeman would immortalise as ‘Metroland’. GCHQ moved to the outer fringes of north-west London, close to Harrow and Pinner. The precise location was Eastcote, which had been used as a wartime outstation of Bletchley Park. It was also close to Dollis Hill, where the laboratories of the Post Office Research Department had built the remarkable ‘Colossus’ computer. Together with Stanmore, Eastcote was one of two large out-stations built in 1943 to accommodate the ever-expanding number of bombes that were being used to cope with the flood of Enigma traffic. However, while it provided reasonable single-storey buildings that were superior to the huts of Bletchley, the overall site was regarded as cramped and unattractive. In June 1946, William Bodsworth, a British code-breaker, returned from a period in America to the cold and rain of an English summer to take over GCHQ’s Soviet section. He found his first sight of Eastcote ‘frankly shattering’. Expecting ‘a nice old country house’, instead he found it to be ‘more cheerless than any of the temporary buildings I have seen in this racket either here or abroad’.62

Those who were leaving Bletchley for good and returning to civilian occupations were given the security warning of their lives. Edward Travis issued a ‘Special Order’ to everyone in GCHQ. He began by thanking them all for their admirable achievements and the substantial contribution they had made to the winning of the war. He then moved quickly on to the matter of maintaining secrecy, even after the end of hostilities. ‘At some future time we may be called upon again to use the same methods. It is therefore as vital as ever not to relax from the high standards of security that we have hitherto maintained. The temptation to “own up” to our friends and families as to what our war work has been is a very real and natural one. It must be resisted absolutely.’63 However, in the Far East, the secret of ‘Magic’, the breaking of Japanese diplomatic codes, was already out. When Bruce Keith, commander of the vast British sigint station located at HMS Anderson in Ceylon, tried to outline Travis’s tight security measures, some of his subordinates openly laughed at him and observed that ‘the Americans had spilled the beans in the paper the other day’.64

The move from Bletchley to Eastcote was undertaken during early 1946 in four main parties. The first was the priority group, and included the Soviet and East European Division; the last arrived in April 1946.65 Staff turning up in leafy Pinner in search of lodgings were allowed to refer to their place of work as ‘GCHQ’, but they were told firmly that any reference to ‘signals intelligence’ was forbidden.66 Between 1945 and 1948 the term ‘GCHQ’ was used interchangeably with both ‘London Signals Intelligence Centre’ and ‘Station X’.67

Bletchley Park was now an empty shell in the Bedfordshire countryside. Barbara Abernethy, who had worked as Denniston’s personal assistant, recalls: ‘We just closed down the huts, put all the files away and sent them down to Eastcote. I was the last person left at Bletchley Park. I locked the gate and took the key down to Eastcote. That was it.’68 Much of the machinery was broken up, including examples of the mighty ‘Colossus’ computational machine. However, Professor Max Newman, who had been central to its development, managed to secure two ‘Colossus’ machines for his new computing department at Manchester University. These were transported by the Ministry of War Transport at the price of thirty-four shillings a ton. Newman offered to send a junior university lecturer down ‘to sit on the van’ to make sure that the precious machines were not damaged in transit.69 In fact, this was not quite the end of Bletchley Park’s active life in sigint, since GCHQ continued to use it for training courses as late as the 1960s.

The intention behind GCHQ’s post-war move to London was to service the centres of power in British government. Accordingly, in the autumn of 1945 Travis took the opportunity to look at how the sigint product – the ‘blue jackets’ or ‘BJs’– circulated around Whitehall. The Foreign Office was a big customer, receiving three sets of BJs daily. One set stayed with Ernest Bevin, the new Foreign Secretary and his war-weary Permanent Under-Secretary, Cadogan, ‘for their immediate information’. Another went to the Services Liaison Department, which worked closely with the JIC. The third went to the main departments. Virtually everyone in the operational core of the Foreign Office habitually saw BJs, but they were always kept separate from other documents in special boxes which were locked up overnight.70

In MI5, the ritual of sigint security was closely observed. Distribution was presided over by the redoubtable ‘Mrs Arbuthnot’, who recorded everything meticulously in her log. Security of BJs seems to have been at its most lax inside SIS, where batches of them circulated around sections for as long as six weeks before being returned. Nor were they properly logged. GCHQ noted that, quite uniquely, inside SIS BJs were never treated as requiring special security measures, and indeed in some cases had ‘found their way into the General Office for filing’. This broke the cardinal rule that sigint was never to mix with ordinary paperwork.71

The first major international crisis of the Cold War era was not long in coming. In June 1948, the Soviets decided to block road and railway access to the western sectors of Berlin, which were controlled by the British, the French and the Americans. The Berlin Blockade was defeated by a massive airlift of some four thousand tons of supplies a day. Hidden amongst the innumerable supply flights heading to Berlin were anonymous but highly secret aircraft collecting sigint for GCHQ, which provided some of the best intelligence during the crisis. Even before the crisis ended in May 1949, GCHQ had already been working hard on the ‘Russian problem’ for almost five years. The early onset of the Cold War had not only provided GCHQ with new targets, but had helped to perpetuate the wartime alliance between British code-breakers and their counterparts in allied countries. This, as we shall see, was fundamental to the postwar success of GCHQ.

4 The KGB and the Venona Project

…Paul [Guy Burgess], and Yan [Anthony Blunt] consider that the situation is serious.

Message from the KGB station in London to Moscow,

February 19501

The ‘Venona Project’ was possibly the most astounding code-breaking effort of the early Cold War.2 Employing perhaps no more than a hundred people, it exploited a weakness in KGB communications and decoded some of the messages sent by Soviet intelligence. As a result, it revealed key Soviet agents and illuminated the unexpectedly vast scope and scale of KGB espionage in the West during the 1940s. This material was so significant that even though no new messages were collected after 1948, British and American code-breakers continued to work on the residue until October 1980. Initiated by the Americans, Venona collected new partners – first the British, and later the Australians, the Canadians, the Dutch and even the ‘neutral’ Swedes. It is justly famous for revealing some of the ‘giants’ of Russian espionage, including Klaus Fuchs and Donald Maclean, but the vast pool of messages that remain unsolved is also significant. Even now, it points unambiguously to many other cases yet to be resolved.

Anxiety about the compromise of sigint secrets was always central to the code-breaking profession. Back in 1927, Prime Minister Stanley Baldwin’s infamous exposure of the reading of Soviet high-grade systems in the House of Commons had taught a whole generation of interwar code-breakers the price of careless talk. Thereafter, anxiety about the Ultra secret persuaded more than ten thousand people to keep their wartime vow of silence for decades. However, Venona introduced an even greater level of paranoia, since it hinted at the possibility of hundreds of Soviet agents active inside the governments of the West, some in high positions. For this reason it is unlikely that Venona was ever made known to President Roosevelt, and it was three years before his successor, Harry Truman, was let into the secret. Clement Attlee, Britain’s first post-war leader, was not told until a major security case made it unavoidable in late 1947.3 Nevertheless, the Venona project was compromised by several Soviet agents within five years of its initiation. This did not entirely negate its value, since the Soviets could not prevent the West from continuing work on the immense volume of KGB messages that had already been collected during the 1940s, patiently revealing the names of important agents. In the late 1950s, for example, GCHQ suddenly began to have success with Soviet Naval Intelligence messages, having used a new analytic technique.4

The extreme secrecy of the Venona project was its Achilles heel. Although the material often pointed to the identity of Soviet spies in the West, for security reasons it could not be shown to those arrested to persuade them to confess; still less could it be produced in court. Any sensible defence lawyer would seek to probe the nature of Venona, not only exposing its fragmentary nature, but also revealing sensitive secrets about sigint. Therefore, once spies had been identified by Venona, they had to be either caught red-handed meeting with their KGB controller, or successfully interrogated and broken. The result was a game of cat and mouse in which the mouse sometimes got away. In 1951, Donald Maclean, Guy Burgess and John Cairncross were among those who escaped by a whisker. Remarkably, Theodore Hall, an American Communist spy within the atomic programme, also brassed it out, despite close interrogation, escaping what would almost certainly have been death in the electric chair.5 By contrast, in 1950 the atom spy Klaus Fuchs succumbed to repeated and patient questioning by MI5 after his arrest. He told his interrogators that he ‘supposed he would be shot’, and was pleasantly surprised when he wasn’t.6

Venona revealed the security-minded nature of the Soviets. Much of their traffic was encrypted using a one-time pad system. This was time-consuming and slow, but they were willing to put in vast effort to protect their communications. This required huge volumes of tear-off pads with sheet after sheet of random numbers. The difficulty of generating thousands of sheets of truly random numbers should not be underestimated, and no one is clear how the Soviets made them. One individual has recalled a room full of women simply shouting out any number that came into their heads, but this seems improbable. Others have described devices not unlike lottery machines, with numbered balls. Whatever system was used, the logistical difficulties of generating many thousands of one-time pads and distributing them proved too much for wartime Russia.7 Some time in early 1942, with Moscow on the verge of evacuation and much of Soviet industry badly dislocated, operators began to run out of pads. The KGB department that printed them committed the fatal error of reprinting twenty-five thousand pages. This made a small proportion of the messages, which should have been unbreakable, vulnerable to cryptanalysis. Far worse, they were sent to KGB units as well as to military and diplomatic users.8

The Venona project that exploited this mistake began in Washington. The Americans had collected Soviet messages during the war, but they lacked time to work on them. On 1 February 1943 the US Army’s code-breaking service, called the Signals Intelligence Service, began a modest effort to see if it could exploit Soviet diplomatic communications. The telegrams had been collected at Arlington Hall, in Virginia, a former girls’ school which was commandeered by the Army as its main code-breaking centre. Interest increased dramatically when it was discovered that some of the streams of traffic related to espionage. In October 1943 a young code-breaker, Lieutenant Richard Hallock, a Signal Corps reserve officer who had been a peacetime archaeologist at the University of Chicago, was looking at Soviet commercial traffic when he realised that the Soviets had committed a terrible error and were reusing their pads. This was an astonishing discovery, and thereafter Venona slowly began to unravel some of the KGB’s most precious secrets.9

The US Army’s head of signals intelligence, Carter W. Clarke, was the main enthusiast for Venona. Clarke was a tough, impatient, hard-drinking individual who many regarded as uncouth, but he was also a lateral thinker. Like many military intelligence chiefs in both Britain and the United States, he nurtured a deep-seated distrust of the Soviets, asserting bluntly: ‘They’re your friends today and they’re your enemies tomorrow, and when they’re on your side find out as much as you can about them because you can’t when they become your enemy.’10 The US Navy code-breakers also began work on Soviet traffic in the summer of 1943. The fact that by the autumn of 1944 the two rival armed services were both referring to all Soviet radio intercepts by the same code name of ‘Rattan’ suggests a directive from a high level. The following year the code name was changed to ‘Bourbon’.11

By 1944, another talented young American code-breaker, Meredith Gardner, was busy making the first breaks into KGB traffic and even some from Soviet military intelligence (GRU). Other code-breakers were now drafted in to help. One of them was Cecil Phillips, a chemistry student who was sent to Arlington Hall in June 1943, initially to work on Japanese naval messages. In May 1944 he was switched to Soviet diplomatic traffic. He quickly realised the scale of duplication, and made a number of progressions that led to wider breaks in the cypher system used by the KGB.

However, substantial activity had to await the end of the war with Japan, when larger numbers of staff could be transferred to work on ‘the Russian problem’.12 Some of the Soviet messages were double-encrypted, and so represented a fantastic level of difficulty. Nevertheless, on 20 December 1946 Gardner decrypted a KGB message listing the names of scientists who had been working on the wartime development of the atomic bomb at Los Alamos, known as the ‘Manhattan Project’. In spring 1947 he decyphered a message that showed that the Soviets were being given highly classified material from inside the US War Department.13 KGB agents were rarely referred to by their real names in the messages. The British spy Donald Maclean, for example, was ‘Homer’ or ‘Gomer’. Accordingly, their identities had to be figured out from their activities and from what material they were providing to the Soviets.14

Early accounts of Venona suggested that the first breaks were achieved as a result of the recovery of a partly burned Soviet codebook found in Finland and sold to America’s wartime intelligence agency, the Office of Strategic Services. Stories have long circulated about how American diplomats insisted that protocol required that it be returned to the Soviets. In fact, up until 1952, the progress made on Venona was probably driven by the pure sweat of mathematics, and represented a remarkable intellectual achievement. A little help was gained by intercepting Japanese traffic that contained Soviet material purchased from the Finns in 1944. The Finns had not been reading high-grade traffic, but had learned enough to be able to sort messages into homogeneous groups, the first stage of a cryptanalytical attack.15 It was only in 1953 that the American team realised that one of the KGB systems it was working on related to a Soviet codebook that had been in their possession since 1945. At the end of the war TICOM Team 6, led by Lieutenant Colonel Paul Neff, had seized a copy of a partially burned Soviet codebook while exploring the German sigint centre at Burgscheidungen. The Germans had themselves seized the codebook from the Soviet Consulate in Petsamo in Finland during June 1941.16

The big shock was revelations about espionage within the Manhattan Project. This immediately raised the question of how the material might be employed for counter-espionage. Liaison was established with Robert Lamphere of the FBI’s Intelligence Division, which had responsibility for maintaining physical surveillance on Soviet espionage activities. Venona was of immense help to the FBI, but it was not a one-way street. Occasionally the Bureau undertook burglaries of Soviet premises and photographed Soviet documents. Over the next decade, attempts were made to match material from these ‘black bag jobs’ with Venona material, but sadly there were few connections. Nevertheless, Lamphere ensured a coordinated exploitation system with the code-breakers.17

Meredith Gardner recalls that tight security for Venona only crept in slowly. In the beginning, everyone in the branch where it was being worked on was potentially privy to it, and ‘no special treatment was given’. This was partly because cryptanalysts had to support each other by discussing problems, since systems were often related to each other. There were people who genuinely needed to know, and there were also ‘mere busy-bodies who perhaps considered themselves consultants at large for all’. The Army intelligence liaison man, Howard Barkley, heard that ‘there was something interesting going on’ and came for a look, even though he had not been formally indoctrinated. Knowledge of Venona ‘might have been picked up almost anywhere’ in the branch at Arlington.18

Yet Venona was ‘so sensational’ that eventually something unusual had to be done on the security front. The focus was less on restricting the knowledge that it existed than on tightly controlling the contents of the messages. However, counter-intelligence is a messy business. What the US Army code-breakers needed in order to identify the spies was background material from other government departments – so they were forced to work closely with a gradually expanding circle of people scattered across Washington. Typically, seven copies of one Venona message, issued on 30 August 1947 and entitled ‘Cover Names in Diplomatic Traffic’, were circulated. One went to GCHQ through its liaison, Colonel Patrick Marr-Johnson. The US Army code-breakers noted that the British surrounded the material with ‘rigid safeguards’. Two copies went to the heads of Army and Navy code-breaking. Four went to mainstream Army Intelligence, Naval Intelligence and FBI. The State Department was also an important collaborator. Given that informal secondary briefing must have taken place, this means that perhaps as many as thirty people may have been given information from one circulated Venona message.19 By contrast, an understanding on Venona was only reached with the CIA in September 1948, and detailed cooperation on active cases did not occur until 1952. Remarkably, this was six years after the American code-breakers had fully indoctrinated the British at GCHQ.20

It was the TICOM raids of early 1945 into Nazi Germany that had forced the British and the Americans to reveal their respective hands on the ‘Russian problem’. TICOM was an Anglo– American project, and no one could disguise the fact that material on German code-breaking successes against the Soviet Union was one of its top priorities. TICOM led to some of the greatest successes of the early Cold War. During the autumn of 1945 some of its best results were coming from a Soviet encyphered teleprinter system code-named ‘Caviar’ which was almost certainly being broken with the help of the German team recovered by Rushworth and Norland on their foray into Germany. No less important was the breaking of a number of Soviet military machine cyphers that were not dissimilar to the Enigma machine, or its widely used Swedish equivalent, the ‘Hagelin’ machine. GCHQ code-named these machines the ‘Poets Systems’. The first success was with an encoded Soviet teletype system code-named ‘Coleridge’ that gave great administrative detail relating to the Red Army in Eastern Europe. Carefully combined with material from more basic techniques such as radio direction-finding, it provided a superbly detailed picture of the Soviet Army in Europe. Thereafter, a team of GCHQ cryptanalysts led by Gerry Morgan working with an American naval team helped to decrypt another Soviet system called ‘Longfellow’. Some of the best successes against Soviet machines were the product of the brilliant mind of Hugh Alexander, combined with the enormous computer power provided by GCHQ’s American allies. In the Far East, Soviet naval codes were beginning to yield, but immediately after the war, ‘Coleridge’ and ‘Longfellow’ were the most important Soviet systems being exploited by the West.21

Britain was told about the embryonic Venona project as early as August 1945, and thereafter John Tiltman, head of the Cryptographic Group at Eastcote, was kept informed of progress.22 However, full cooperation came a little later. The young American code-breaker Cecil Phillips spent six months at GCHQ’s new location at Eastcote collaborating with Philip Howse. They focused on Soviet traffic that had been collected in Australia by monitoring Moscow’s Embassy in Canberra. More senior figures such as John Tiltman did not give them much attention, since Phillips and Howse initially thought much of the traffic to be low-level consular material. In 1947 GCHQ received a further briefing, this time from Meredith Gardner, the key American analyst of the Venona messages. However, GCHQ did not set up a proper Venona office at Eastcote until December 1947, sparked by the recognition that the Australian material was actually KGB traffic.23 Eastcote was itself in a state of permanent revolution, with sections being constantly reformed and merged, to the extent that the ‘rumblings of reorganisation’ drew comment from figures like Joseph Wenger, Washington’s senior naval code-breaker.24 The rumblings were the sounds of growth. From an establishment of just over a thousand in December 1945, GCHQ was nudging three thousand staff by 1948, and was already looking for new premises to accommodate its swelling numbers.25

The British had also collected plenty of interesting wartime KGB traffic. As early as June 1943, Alastair Denniston had met Colonel Ted Maltby of the Radio Security Service, together with Roger Hollis and John Curry of MI5, to discuss ‘the interception of certain apparently illicit transmissions from this country which have been “DF-ed” to the Soviet Embassy’. (‘DF’ referred to the technique of radio direction-finding by triangulating between several aerials, sometimes mounted on detector vans.) These messages had attracted interest because they had nothing in common with the old Comintern style of transmissions, and it was noted that they might well be KGB traffic as they showed ‘great technical skill’. Collecting this material stretched Britain’s interceptor resources, since the traffic had lasted for eight hours solid in every twenty-four-hour period. Meanwhile, it was also searching for an illegal Comintern radio station in Wimbledon, using a disguised Ford Thames van with direction-finding equipment and security personnel in civilian clothes.26

By 1948, the Venona teams at GCHQ and Arlington Hall were small but extremely integrated. Although the British employed a different code name for Venona, calling it ‘Bride’, they adopted a standard procedure for the translations. The British cell was superintended by William Bodsworth, one of the initial team that began studying Enigma in 1937.”27 Like so many interwar code-breakers, Bodsworth was a linguist, not a mathematician, having read Spanish at Cambridge. Cheerful and possessed of a gentle humour, he was dubbed ‘Snow White’ because of his mop of white hair. Bodsworth’s team undertook much of the laborious task of trying to reconstruct the Soviet codebooks. The seven dwarfs supplied almost enough nicknames for the Venona teams: by the end of 1950, the number of people at Eastcote working on ‘Bride’ remained at less than ten. For the Americans, British input was essential both to the efforts to track down the identity of figures like ‘Homer’ and to obtaining background material to allow the analysis of the KGB’s Canberra messages.28

It is almost certain that the first person to alert the Soviets to the existence of the Venona project in any detail was a KGB agent named William Weisband. Born in the Soviet Union in 1908, Voldya Weisband had emigrated with his family to the United States in the 1920s. In 1940 he had changed his name to William, and had registered at the American University in Washington DC. By 1942 he was serving as a lieutenant in a US Army code-breaking unit in the Middle East. He was posted back to Arlington Hall in July 1944, and was soon working in the Soviet section. Weisband had in fact been a KGB agent since 1934, and he certainly displayed all the traits of a classic agent. Gregarious and popular, he had friends throughout what was now called the Army Security Agency, and charmed the senior officers. His reputation as a problem-solver allowed him wide access within the Soviet section, and Meredith Gardner actually recalls him looking at a list of names derived from Venona material in late 1946. Weisband was not himself identified by Venona, but seeing the messages decrypted must had made him feel queasy, since his name – or at least his code name ‘Zhora’ – was certainly buried in traffic somewhere. In 1948 the Soviets summarised Weisband’s reports that had been fed back to KGB headquarters in Moscow. They contained worrying news: