The Secret War: Spies, Codes and Guerrillas 1939–1945

The Japanese spied energetically in China, the US and the European South-East Asian empires, which they viewed as prospective booty. Their agents were nothing if not committed: in 1935 when police in Singapore arrested a local Japanese expatriate on suspicion of espionage, such was the man’s anxiety to avoid causing embarrassment to Tokyo that he followed the E. Phillips Oppenheim tradition and swallowed prussic acid in his cell. The Chinese Nationalists headed by Chiang Kai-shek sustained an effective counter-intelligence service to protect his dictatorship from domestic critics, but across Asia Japanese spies were able to gather information almost unhindered. The British were more interested in countering internal communist agitation than in combating prospective foreign invaders. They found it impossible to take seriously ‘the Wops of the East’, as Churchill called the Japanese, or ‘the little yellow dwarf slaves’, in the words of the head of the Foreign Office.

Britain’s diplomats were elaborately careless about protecting their secrets, adhering to the conventions of Victorian gentlemen. Robert Cecil, who was one of them, wrote: ‘An embassy was an ambassador’s house party; it was unthinkable that one of the guests could be spying on the others.’ As early as 1933 the Foreign Office received a wake-up call, albeit unheeded: after one of its staff put his head in a gas oven, he was revealed to have been selling British ciphers to Moscow. Next a clerk, Captain John King, was found to have been funding an American mistress by peddling secrets. In 1937 a local employee in Britain’s Rome embassy, Francesco Constantini, was able to rifle his employer’s papers for the benefit of the Italian secret service, because the ambassador assumed that one could trust one’s servants. At that period also, Mussolini’s men read some British codes: not all Italians were the buffoons their enemies supposed. In 1939, when Japanese intelligence wanted the codebooks of the British consulate in Taipei, its officers easily arranged for a Japanese employee to become night-duty man. During the ensuing six months Tokyo’s agents repeatedly accessed the consulate safe, its files and codebooks.

Yet nowhere in the world was intelligence wisely managed and assessed. Though technological secrets were always useful to rival nations, it is unlikely that much of the fevered secret political and military surveillance told governments more than they might have gleaned from a careful reading of the press. Endemic rivalries injured or crippled collaboration between intelligence agencies. In Germany and Russia, Hitler and Stalin diffused power among their secret policemen, the better to concentrate mastery in their own hands. Germany’s main agency was the Abwehr, its title literally meaning ‘security’, though it was responsible for both intelligence-gathering abroad and counter-espionage at home. A branch of the armed forces, it was directed by Admiral Wilhelm Canaris. When Guy Liddell, counter-espionage director of MI5 and one of its ablest officers, later strove to explain the Abwehr’s incompetence, he expressed a sincere belief that Canaris was in the pay of the Russians.

The Nazis also had their own security machine, the Reichssicherheitshauptamt or RSHA, directed by Ernst Kaltenbrunner within the empire of Himmler. This embraced the Gestapo secret police and its sister counter-intelligence branch the Sicherheitdienst or SD, which overlapped the Abwehr’s activities in many areas. A key figure was Walter Schellenberg, Reinhard Heydrich’s aide: Schellenberg later took over the RSHA’s foreign intelligence-gathering service, which subsumed the Abwehr in 1944. High Command and diplomatic codebreaking activities were conducted by the Chiffrierabteilung, colloquially known as OKW/Chi, and the army had a large radio intelligence branch that eventually became OKH/GdNA. Göring’s Air Ministry had its own cryptographic operation, as did the Kriegsmarine. Economic intelligence was collected by the WiRuAmt, and Ribbentrop’s Foreign Ministry gathered reports from embassies abroad. Guy Liddell wrote crossly: ‘Under our system of government there was nothing to stop the Germans from getting any information they required.’ But the elaborate Nazi intelligence and counter-espionage machines were far more effective in suppressing domestic opposition than in exploiting foreign sources, even when they heard something useful from them.

France’s intelligence departments enjoyed a lowly status and correspondingly meagre budgets. Pessimism overlaid upon ignorance caused them consistently to overstate German military strength by at least 20 per cent. František Moravec believed that politics crippled French security policy as war loomed: ‘Their desire to “know” seemed to decrease proportionately as the Nazi danger increased.’ Moravec the Czech found his French counterparts half-hearted colleagues, though he returned from one inter-Allied conference with a present from a famous French criminologist, Professor Locarde of Lyons: a chemical developer which proved useful for exposing secret writing.

Since the beginning of time, governments had been able to intercept each other’s communications only when spies or accidents of war physically diverted messages into their hands. Now, however, everything was different. Wireless communication was a science slightly older than the twentieth century, but thirty years elapsed before it became a universal phenomenon. Then, during the 1930s, technological breakthroughs prompted a global explosion of transmissions. The ether hummed, whined and crackled as messages private, commercial, military, naval, diplomatic traversed nations and oceans. It became indispensable for governments and their generals and admirals to communicate operational orders and information by radio, to every subordinate, ship and formation beyond reach of a landline. Making such exchanges secure demanded nice judgements. There was a trade-off between the speed at which a signal could be dispatched and received, and the subtlety of its encryption. It was impracticable to provide front-line army units with ciphering machines, and thus instead they employed so-called hand- or field-ciphers, of varying sophistication – the German army used a British-derived system called Double Playfair.

For the most secret messages, the only almost unbreakable code was that based upon a ‘one-time pad’, a name that reflected its designation: the sender employed a unique combination of letters and/or numbers which became intelligible only to a recipient pre-supplied with the identical formula. The Soviets especially favoured this method, though their clerks sometimes compromised it by using a one-time pad more than once, as the Germans found to their advantage. From the 1920s onwards, some of the major nations started to employ ciphers which were deemed impregnable if correctly used, because messages were processed through electrically-powered keyboard machines which scrambled them into multi-millions of combinations. The magnitude of the technological challenge posed by an enemy’s machine-encrypted signals did not deter any nation from striving to read them. This became the most important intelligence objective of the Second World War.

The brightest star of the Deuxième Bureau, France’s intelligence service, was Capitaine Gustave Bertrand, head of the cryptanalytical branch in the army’s Section des Examens, who had risen from the ranks to occupy a post that no ambitious career officer wanted. One of his contacts was a Paris businessman named Rodolphe Lemoine, born Rudolf Stallman, son of a rich Berlin jeweller. In 1918 Stallman adopted French nationality; simply because he loved espionage as a game in its own right, he began to work for the Deuxième. In October 1931 he forwarded to Paris an offer from one Hans-Thilo Schmidt, brother of a German general, to sell France information about Enigma in order to dig himself out of a financial hole. Bertrand accepted, and in return for cash Schmidt delivered copious material about the machine, together with its key settings for October and November 1932. Thereafter he remained on the French payroll until 1938. Since the French knew that the Poles were also seeking to crack Enigma, the two nations agreed a collaboration: Polish cryptanalysts focused on the technology, while their French counterparts addressed enciphered texts. Bertrand also approached the British, but at the outset they showed no interest.

Britain’s codebreakers had acquired an early-model commercial Enigma as early as 1927, and examined it with respect. Since then, they knew that it had been rendered much more sophisticated by the inclusion of a complex wiring pattern known as a Steckerbrett, or plugboard. It now offered a range of possible positions for a single letter of 159 million million million. That which human ingenuity had devised, it was at least theoretically possible that human ingenuity might penetrate. In 1939, however, no one for a moment imagined that six years later intelligence snatched from the airwaves would have proved more precious to the victors, more disastrous for the losers, than every report made by all the spies of the warring nations.

2 THE BRITISH: GENTLEMEN AND PLAYERS

The reputation of MI6 was unmatched by that of any other secret service. Though Hitler, Stalin, Mussolini and Japan’s generals shared a scepticism, or even scorn, about the old lion’s fitness to fight, they viewed its spies with extravagant respect, indeed cherished a belief in their omniscience. British prowess in clandestine activity dated back to the sixteenth century at least. Francis Bacon wrote in his History of the Reign of King Henry VII: ‘As for his secret Spials, which he did employ both at home and abroad, by them to discover what Practises and Conspiracies were against him, surely his Case required it.’ Queen Elizabeth I’s Sir Francis Walsingham was one of history’s legendary spymasters. Much later came the romances of Rudyard Kipling’s Kim, of John Buchan’s Richard Hannay, of dashing ‘clubland heroes’ who played chess for England with a thousand live pieces across a board that spanned continents. A wartime British secret servant observed: ‘Practically every officer I met in that concern, at home and abroad, was, like me, imagining himself as Hannay.’ The great Danish physicist Niels Bohr told the scientific intelligence officer R.V. Jones that he was happy to cooperate with the British secret service because ‘it was run by a gentleman’.

British intelligence had enjoyed a good Great War. The Royal Navy’s codebreakers, such men as Dillwyn Knox and Alastair Denniston, labouring in the Admiralty’s Room 40, provided commanders with a wealth of information about the motions of the German High Seas Fleet. The decryption and public revelation of Berlin’s 1917 Zimmermann Telegram, urging the Mexicans to take aggressive action against the United States, played a critical role in bringing the Americans into the war. For two years after the November 1918 Armistice, the secret service was deeply involved in the Allies’ unsuccessful attempt to reverse the outcome of the Russian Revolution. Even after this was abandoned, the threat from international communism remained the foremost preoccupation of British espionage and counter-espionage.

Yet amid the inter-war slump, funding was squeezed. MI6 mouldered, to an extent little understood by either Britain’s friends or foes. Hugh Trevor-Roper, the historian who became one of its wartime officers, wrote: ‘Foreign intelligence services envied the British secret service; it was their idealised model … It enjoyed the reputation of an invisible, implacable force, like the Platonic world-spirit, operating everywhere. To the Nazi government, it was at the same time a bogey and an ideal … The reality … was rather different.’ MI6’s senior officers were men of moderate abilities, drawn into the organisation by the lure of playing out a pastiche of Kipling’s ‘Great Game’, and often after earlier careers as colonial policemen.

They masqueraded as passport control officers in embassies abroad, or shuffled paper in the service’s austere – indeed, frankly squalid – headquarters beside St James’s Park underground station, in Broadway Buildings, a place of threadbare carpets and unshaded lightbulbs. MI6 sustained a quirky tradition of paying its staff tax-free and in cash, but so small a pittance that a private income was almost essential for officers who aspired to an upper-middle-class lifestyle, which meant all of them. Though its budget was progressively increased from £180,000 in 1935 to £500,000 in 1939, few graduates entered the service, because its bosses did not want them. MI6, in the view of one practitioner, was designed merely to receive intelligence rather than actively to procure it. It was run by a coterie of anti-intellectual officers who saw their principal, if not sole, task as that of combating revolutionary communism. The shift of emphasis to monitoring Nazis and fascists during the late pre-war period caused great difficulties.

Some recruits of that period proved ill-suited to the essential nastiness of espionage. Lt. Cmdr Joseph Newill, a retired sailor posted to Scandinavia in 1938 on the strength of speaking Norwegian, wailed to London: ‘I doubt whether I have the natural guile so essential for this work!’ Newill complained that his role involved much more hard labour than he had expected. He told his station chief petulantly: ‘I am 52 and I am not going to work myself to death at my time of life.’ But he was kept in the job, and contrived to meet Broadway’s undemanding standards. MI6’s Shanghai station chief, Harry Steptoe, operated under cover as vice-consul. A jaunty little cock-sparrow figure who affected a moustache and monocle, he puzzled a foreign diplomat by his appearance at receptions in a lovat-green suit adorned with gold braid. Was this, demanded the diplomat, the full-dress uniform of the British secret service? When the Japanese interned Steptoe in 1942, they dismissed the possibility that such a comic figure could be a spymaster, and instead subjected to brutal interrogation a hapless British Council representative, whose field of knowledge was exclusively cultural.

Broadway struggled to secure intelligence from the Continent. In 1936 a new MI6 department was formed to monitor Germany and Italy. Z Section was run by Claude Dansey, a former imperial soldier who bore a haversack groaning with blimpish prejudices, among them a loathing for Americans. It became an almost independent fiefdom, which operated under commercial cover from offices in Bush House in The Strand. Its sources were mostly elderly retreads such as the Lithuanian Baron William de Ropp, who for more than a decade extracted from the British £1,000 a year – a handsome competence – in return for fragments of German political gossip. The Nazis were well aware of de Ropp’s role, and fed him what they wanted London to hear. In August 1938 the Baron decided that his secret life had become too fraught, and wisely retired to Switzerland.

Naval engineer Dr Karl Kruger’s story had a darker ending. From 1914 to 1939 he fed some good information to the British on a cash-and-carry basis, but vanished from sight a month before the outbreak of war. His file at Broadway was eventually marked ‘Agent presumed “dead”.’ This was not surprising, because Kruger – like most of MI6’s German informants – was controlled by its Hague station, where one of the local staff, Folkert van Koutrik, was on the Abwehr’s payroll. The service’s best pre-war humint source was Wolfgang Gans Edler zu Putlitz, press attaché at the German embassy in London, an aristocrat and homosexual. He was run by Klop Ustinov – father of the actor Peter – a Russian-born journalist who lost his newspaper job in 1935 because of his Jewishness. When Putlitz was transferred to The Hague in 1938, Ustinov followed him at MI6’s behest. After Folkert van Koutrik later betrayed the British operation in Holland, Putlitz hastily sought asylum in London.

The flow of intelligence from the Continent was thin. The Air Ministry complained about the paucity of material on the use of aircraft in the Spanish Civil War, an important issue for planners. Britain’s ambassador in Berlin, Sir Nevile Henderson, shared with his fellow-diplomats a disdain for espionage which caused him to refuse diplomatic status to Broadway’s ‘Passport Control Officers’. Even where MI6 tried to provide German informants with wireless sets, most were reluctant to take them, because discovery of such equipment by the Gestapo ensured a death sentence for the possessor.

Very occasionally, among the mountain of rubbish that accumulated in Broadway’s files there was a pearl. In the spring of 1939 an agent codenamed ‘the Baron’, with good social connections in East Prussia, reported to his handler Harry Carr in Helsinki that the Germans were secretly negotiating with Stalin. He followed this up with a further missive in June, asserting that talks between Berlin and Moscow were making good progress. Yet this sensational pointer to the looming Nazi–Soviet Pact, which afterwards proved to have come from gossip among aristocrats working in the German Foreign Ministry, was dismissed in Broadway. To MI6’s senior officers, a devils’ pact between Stalin and Hitler seemed a fantastic notion. An authentic scoop was missed; first, because MI6, like most intelligence organisations, had an instinctive and usually prudent scepticism about its own sources; second, because what ‘the Baron’ reported ran contrary to his employers’ expectations. At that time, and indeed throughout the war, MI6 had no internal machinery for analysing incoming intelligence, though its chiefs could point out that the Axis Powers lacked this also.

Czechoslovakia and Poland occupied the front line in the European confrontation with Hitler. MI6 showed little interest in collaboration with their intelligence services until March 1939, when the strategic picture changed dramatically: the British and French governments gave a security guarantee to Poland. This galvanised Broadway.

On 25 July, a British delegation composed of a naval intelligence officer together with Alastair Denniston, director of the Government Code & Cypher School, and Dillwyn Knox, one of its foremost codebreakers, joined France’s Gustave Bertrand – himself no cryptographer, but a notable facilitator and diplomat – at an exploratory meeting with their Polish counterparts led by Col. Gwido Langer, held at their cryptographic centre in the Kabackie woods near Pyry, south of Warsaw. The first day’s talks, conducted in mixed French and German, went very badly. Knox, for reasons unknown, was in a vile temper, and highly sceptical that the Poles had anything to tell worth hearing. He seemed unable to understand the methods by which they claimed to have achieved the breakthrough which had enabled them to read some German naval traffic. All the parties present were fencing, to discover each other’s state of knowledge. Warsaw’s decision to involve the British was prompted by new difficulties that had frustrated their own codebreakers since the Germans on 1 January adopted an enhanced stecker board, for their Enigmas, with ten plugs instead of seven. On the second day, 26 July, the conference’s atmosphere was transformed for the better. In the basement of the building the Poles showed off their ‘bomby’, primitive computing devices designed to test multiple mathematical possibilities. Then they produced a coup de théâtre: they presented both visiting delegations with mimicked copies of the Enigma built by their own men. Knox’s scepticism crumbled, and the meeting ended in a mood of goodwill and mutual respect. Everybody at Broadway recognised the importance of the Poles’ gesture to their allies as a contribution to the secret struggle against the Nazis. Marian Rejewski, a former mathematics student at Warsaw University who had joined the Kabackie woods team back in 1932, is today acknowledged as a pioneer among those who laid bare the secrets of Enigma, even if it fell to others, in Britain, to advance and exploit Rejewski’s achievement.

Stewart Menzies, then deputy chief of MI6, was so impressed by the outcome of the Polish trip that he turned up in person at Victoria station to greet Gustave Bertrand – and to inspect the mimicked Enigma. Knox sent the Poles a gift of scarves, decorated with images of Derby runners, with the letter thanking his hosts for their ‘co-operation and patience’. At or around this time also, the Poles provided the British with five of the Enigma’s eight alternative rotors. A chasm still yawned, however, between understanding how the machine worked, and achieving the ability to read its traffic. Though a trickle of German messages were broken by human ingenuity during the winter of 1939–40, traffic was breached on an industrial scale only from 1941 onwards, following the creation of revolutionary electro-mechanical technology. Nonetheless, the assistance of the French and Poles dramatically accelerated progress at the GC&CS, now evacuated from London to a safer country home. Physical possession of the enemy’s encryption instrument enabled its cryptanalysts to grasp the mountainous challenge they must overcome.

Until 1939, and in large measure for two years thereafter, British intelligence remained dependent for its view of the world upon humint – reports from informants abroad. How well did MI6 fulfil its responsibility to brief the government about the mounting threat from Nazi Germany – ‘Twelveland’ in Broadway parlance? It produced many reports arguing that Hitler’s long-term ambitions lay in the East, and this was fundamentally correct. Unfortunately for its credibility, however, in 1940 Germany chose first to seek to dispose of the Western democracies. MI6 was in no doubt that Hitler was rearming fast, but insistently emphasised the weakness of the industrial base from which he aspired to make war. Responsibility for gathering economic data rested with the Industrial Intelligence Centre, an offshoot administered since 1934 by the Foreign Office, but run by the veteran secret service officer Major Desmond Morton. During the ‘wilderness years’, Morton passed to Winston Churchill – with the sanction of prime minister Stanley Baldwin – details of German rearmament which empowered the unheeded prophet to cry forth warnings to the world. Ironically, the Major wildly overstated the growth of Hitler’s military machine: Morton never had much grasp of economics in general, nor of the Nazi economy in particular.

But modern historians critical of pre-war British intelligence failures miss some important points. In those days few people of any nationality understood economic analysis. The IIC was correct in judging that Germany was ill-prepared to conduct a long struggle, and was rendered vulnerable by its dependence on imported commodities and especially oil. The German economy, as Adam Tooze has shown, was not strong enough to meet the huge challenge Hitler sought to fulfil, of conquering the most advanced societies on earth. Germany’s GDP was no larger than Britain’s, and her people’s per capita incomes were lower. In 1939, Hitler’s expenditures on armaments had reduced his country’s finances to a parlous condition. But it was asking too much of any intelligence service to gauge the potential of German industry under the stimulus of conflict: to the very end of World War II, the best brains in the Allied nations failed fully to achieve this. MI6 could not be expected to predict Hitler’s conquests, which dramatically enhanced his access to oil, raw materials and slave labour.

On the military side, neither MI6 nor the service departments learned much about the new technology and tactics being developed by Britain’s enemies. Nor about their limitations: they wildly overrated the Luftwaffe’s ability to devastate Britain’s cities. In 1938, Broadway reported that the Germans had 927 first-line bombers capable of mounting 720 sorties a day and dropping 945 tons of ordnance (this was an exaggeration of 50 per cent), and projections of likely casualties were even more inflated. War Office appreciations of the German army were equally mistaken, especially in estimating its potential mobilised strength. These suggested in 1939 that Hitler was already master of the largest war machine his nation’s resources could bear. Rearmament, coupled with vast public expenditure, ‘had taxed the endurance of the German people and the stability of the economic system to a point where any further effort can only be achieved at the risk of a breakdown of the entire structure’.