The Black Door: Spies, Secret Intelligence and British Prime Ministers

No prime minister illustrates this interface between premiers and personal hazard more than Churchill. Britain’s wartime leader accepted the advice of the Special Operations Executive that assassinating Hitler would be counter-productive, since his strategy was increasingly incompetent and damaging to Germany. However, in 1943 Churchill did approve an assassination attempt against Mussolini, with female agents being dropped into Italy, together with aggressive kill missions against Rommel and other senior German generals. Meanwhile, lesser fascists were earmarked for mere bribery, with an initial £100,000 personally approved by Churchill and then passed over in a large bag to members of Franco’s circle on a Spanish golf course as an inducement to keep Spain out of the Second World War.36 Eden was appalled by the operation, but when more money was required for Franco, Churchill eagerly minuted in red ink: ‘Yes Indeed. WSC.’37

Britain’s wartime leader was oddly relaxed about his own personal safety, despite more than a dozen serious attempts on his life. Hitler certainly ordered his multiple secret services to target Churchill, the oddest such operation being a plot to attack Allied leaders at the Tehran Conference in 1943 using a team of Nazi agents transported by camel.38 Much of Churchill’s remarkable ability to survive the attempts on his life must be attributed to good fortune and his ever-vigilant bodyguard, Detective Inspector Walter Thompson. Occasionally Churchill insisted on carrying his own revolver, but for the most part brazen and carefree with only a single bodyguard, he travelled over 200,000 miles across Europe, America and the Middle East in the course of the war. No British prime minister will ever do that again, so in many ways Churchill stands at the turning point of security and intelligence in Downing Street, representing both the first of a new wave of premiers who exploited intelligence properly, but also the last of a dying breed.

Twenty prime ministers have led Britain since 1909. From Herbert Asquith to Theresa May and from Ramsay MacDonald to Gordon Brown, these premiers have held diverging political views, possessed wildly different leadership styles, and have confronted the full spectrum of problems, threats and crises. This book examines each prime minister in turn. It traces their personal approaches to intelligence and uses each premiership as a vehicle to explore the most pressing national security issues of the day. It reveals that despite the vagaries of personality and politics, intelligence has become increasingly central to all prime ministers. This was not always the case. Back in the first decade of the twentieth century, the secret world was alien to Asquith. Churchill and Attlee addressed this problem together, and forged a quiet revolution. Now, as a result, the prime minister presides personally over an organised intelligence community operating at the heart of Whitehall. This book traces that secret service journey from the periphery to the centre of power.

PART ONE

CREATING SECRET SERVICE

1

Herbert Asquith, David Lloyd George and Andrew Bonar Law (1908–1923)

A rumour spread next day in true war-time fashion that Lord Chetwynd had caught three German spies trying to signal the Zeppelin with lights, and had shot them out of hand.

David Lloyd George1

The British intelligence system was already immense by the dawn of the twentieth century. The entire British Empire, sprawling across more than half the globe, served as an intelligence machine – a veritable ‘empire of intelligence’.2 It depended on information of every kind to project British rule in remote places. Romanticised in the works of Rudyard Kipling, intelligence was about amateur imperial adventurers, revelling in deceit and subterfuge. Unlike today, espionage was conducted informally by local agents, eccentric travellers and scholars. Lawrence of Arabia and the legendary team of British archaeologist-spies at Carchemish, working across the Turkey­–Syria border, typified the way great-power rivalry in the Middle East and Asia mixed with academic debates about how archaeology might change understandings of the Bible. Importantly, scholarly research was not merely a cover for secret service work. Intelligence was considered to be every kind of information, including that obtained by peculiar scholars who spoke many languages, collected eggs, bulbs and butterflies, and mixed ‘comfortably and innocently’ with the local population. Indeed, at the turn of the twentieth century, British officials in Mesopotamia feared an archaeologist-gap and urged the India Office to contact learned societies which could send more scholar-spies to buttress British interests.3 India itself, the vast jewel of the empire, depended on spies too. It became an extraordinary ‘empire of information’, allowing just over a thousand British civil servants to superintend close to 280 million people.4

An empire of information, in some form, existed at home too. New public health initiatives and social programmes meant the state knew much more about its people than ever before. Meanwhile Irish nationalists and bomb-throwing anarchists from the Continent had forced the Metropolitan Police to develop the world’s first ‘Special Branch’ in 1883. Inevitably perhaps, by the first decade of the twentieth century, Special Branch had broadened its net beyond Fenians and anarchists to collect intelligence on possible foreign spies, anti-colonial agitators, suffragettes, union leaders, pacifists, and even those with radical views on sex.5

Nevertheless, these developments were more about keeping authoritarian surveillance at arm’s length from the British public. Mysterious battles with Russian spies in Central Asia were reassuringly remote, while the creation of a small Special Branch of intrepid detectives to counter terrorism was seen as a low-key alternative to the sort of harsh and repressive security legislation preferred in Europe. Accordingly, intelligence remained a matter for distant vice-consuls travelling to Samarkand, or detective sergeants in Whitechapel. It rarely resonated at the centre of British government. More rarely still did a prime minister take a personal interest in espionage.

All this began to change over the next decade. Bizarrely, it was literary fashion which drove the transformation of British intelligence, thrusting it towards the heart of government for the first time. A wave of popular Edwardian ‘invasion literature’ forced Herbert Asquith, an otherwise uninterested Liberal prime minister more focused on progressive domestic reforms, to take notice of foreign espionage. Acting only under pressure, Asquith set in train a course which would fundamentally alter the landscape of British intelligence forever. The intelligence revolution may have been forged by Churchill and Attlee, but the first sparks came through the unlikely Asquith. Nonetheless, despite the expansion of espionage during the First World War and greater interest on the part of Asquith’s successor David Lloyd George, it took time and endless trouble before successive prime ministers learned to use intelligence effectively.

Fiction paved the long pathway to the creation of a modern British intelligence service. During the nineteenth century, a stream of often sensational crime novels fed a growing public fascination with detection, police work and science. This generated cultural change, gradually eroding anxieties about government surveillance. Created by writers as diverse as Edgar Allan Poe and Arthur Conan Doyle, fictional detectives gradually became heroic figures to be worshipped, rather than bogeymen associated with distasteful and un-British authoritarianism.6 Journalists, who often exchanged information with real policemen, accelerated this transformation, helping to create real-life detectives who enjoyed a similar cult status.7 William Melville, who became head of Special Branch, was one example. He famously foiled several anarchist attacks, including the ‘Jubilee plot’ against Queen Victoria in 1887 and another dastardly scheme known as the ‘Walsall plot’, in which anarchists from the West Midlands sought to manufacture a bomb. Many now believe that he created the latter plot himself for mere self-glorification.8

Over three hundred spy novels went into print between 1901 and 1914, mostly focused on the ‘German Menace’. This new fiction effectively rebranded itself as a barely concealed form of ‘true crime’ writing, supposedly based on the patriotic leaking of government secrets. In 1903, Erskine Childers published The Riddle of the Sands, which narrated the summer sailing adventures of two young British men along the East Frisian coast who stumble upon a German plot to invade England with a flotilla of barges. Childers claimed to be offering the British people nothing less than a warning of what was to come, and Rudyard Kipling urged the public to support him in taking a firm stand against the ‘shameless Hun’.9 The story was improbable – the German admiralty had long written off the possibility of an invasion from East Frisia – but the hapless Royal Navy was forced to investigate the claims regardless.10

Members of Parliament from constituencies on the east coast, the closest part of Britain to Germany, bombarded the First Lord of the Admiralty, Lord Selborne, with difficult questions. In response, he ordered the Naval Intelligence Division to write a detailed report on the invasion plan outlined by Childers. After reconnaissance of the Frisian coast, the report concluded that the lack of railways and roads, together with the shallowness of the water and general lack of facilities, made a secret invasion from there impossible. ‘As a novel it is excellent; as a war plan it is rubbish,’ insisted Lord Louis Battenberg, the director of naval intelligence. To Childers’ barely disguised delight, the Kaiser banned the book in Germany. Childers also claimed that when he next went sailing in the Baltic, German spies dogged his every move.11

Spy fiction developed a darker side, shifting its focus closer to home, portraying immigrants and foreign visitors as the hidden hand of subversion. The new trope was Germans as a hidden fifth column already within Britain. Most active of all was a mischievous thriller-writer called William le Queux. He almost single-handedly created a fifth-column panic and then demanded the creation of a domestic security service to combat the undercover ‘German Menace’, with which the day of reckoning would surely come.12 Le Queux’s The Invasion of 1910 became the most influential of these books. Published in 1906, it told of German armies over-running the country with the help of spies and saboteurs. Le Queux admitted to a political purpose, explaining his intention to illustrate how poorly British defences would stand up to this sort of sneak attack. He also insisted that the content of his book was factually correct, and had been informed by conversations with ‘the authorities’.13

William le Queux posed as an international espionage expert, but was in fact nothing of the sort. Instead, he was a tireless and well-paid pulp-fiction writer who produced five novels a year until his death in 1927.14 The Invasion of 1910 sold more than a million copies, and publicists at the Daily Mail, which possessed serialisation rights, sent columns of men marching up Oxford Street dressed in Prussian uniforms, complete with bloodstained gloves, carrying sandwich boards that advertised the book.15 An inevitable flood of literary emulators hit the presses as le Queux’s royalties rolled in. Not to be outdone, le Queux responded in 1909 with Spies of the Kaiser, which insisted that no fewer than 5,000 German undercover operatives were at work in Britain. This was a new idea. A whole undercover army, not just a few spies and saboteurs, were apparently biding their time until the Fatherland told them to retrieve their weapons from a series of arms dumps in the British countryside. No less important, the novel also claimed to offer the inside story of intrepid British detectives working with agents to uncover these foreign networks.16

The British public were whipped into a frenzy. Politicians who wanted to expand Britain’s relatively small peacetime army jumped on the bandwagon. In 1908 Lord Roberts, Britain’s most distinguished soldier, claimed there were 80,000 trained undercover German soldiers in England ready to assist in the event of an invasion.17 Newspapers offered £10 to members of the public who reported sightings of suspicious activities that they could pass on to le Queux so he could ‘supplement his investigations’. Unsurprisingly, they were inundated with information, and soon it seemed a supposed German saboteur had been spotted in every town in the land,18 including some unfortunate Foreign Office clerical staff holidaying on the east coast.19 The government was annoyed by the obvious political scheming by advocates of increased arms spending. In 1906, Asquith’s predecessor as prime minister, Henry Campbell-Bannerman, rose to his feet in Parliament to denounce le Queux as a ‘pernicious scaremonger’ whose stories risked provoking an unnecessary war with Germany.20

Despite prime ministerial condemnation, several key figures in government worked hand in glove with le Queux. James Edmonds, head of a fledgling military intelligence unit called MO(5), maintained that Berlin had an extensive ring of operatives in Britain. Edmonds had long been nagging a dismissive Richard Haldane, secretary of state for war under both Campbell-Bannerman and Asquith, about the shortcomings of British counter-espionage. He therefore found le Queux’s activities more than welcome.21 Indeed, much of the evidence that he presented to his masters came from le Queux, who in turn said that he had received it from concerned members of the public. There had only been five cases of German espionage reported in 1907, but unsurprisingly by 1908 this had risen to forty-eight. Edmonds created a helpful map of agent sightings. Perhaps equally unsurprisingly, this was soon leaked to the press. He also recruited William Melville, the former head of Special Branch, who while a talented agent-runner, had a reputation for embellishing his stories.22

By 1909, nefarious German agents were apparently hiding behind almost every bush in Britain. Asquith, however, had other things on his mind. His overwhelming, and rather challenging, priority was to take on and reform the House of Lords, which had consistently blocked his progressive agenda. The time and energy he devoted to this issue only increased further when the Lords famously rejected his government’s budget in 1909. But with the public suffering from spy-fever, the prime minister now had to take note of intelligence matters. As chancellor of the exchequer and in his early months as prime minister, Asquith had already chaired a senior committee to consider the invasion threat in response to pressure from Roberts and le Queux. His report demolished their theories, and showed a surprise attack to be impossible.23 Nonetheless, still under immense public pressure, in 1909 Asquith asked the Committee of Imperial Defence to consider the danger of foreign agents operating in the UK. All this accelerated an important cultural change away from the idea that counter-espionage at home was authoritarian and un-British.24

Although flawed and bogus intelligence reports shaped its ultimate findings, Asquith swiftly sanctioned the committee’s recommendation to establish a new Secret Service Bureau. This top-secret decision – very few people knew about the bureau’s existence – fundamentally altered the landscape of British intelligence forever. British espionage was nothing new, but its formalisation and growing proximity to Downing Street marked the beginning of a more organised and centralised intelligence system. It was not yet, though, an intelligence community linked to Downing Street, and early prime ministers remained unable or unwilling to engage with the secret world particularly closely.

The Secret Service Bureau was comprised of two branches. MO5(g) was the domestic branch, responsible for counter-espionage, and would soon come to be known as ‘MI5’, or the security service. It inherited a number of army counter-intelligence officers, and was led by Captain Vernon Kell, deputy to James Edmonds at MO(5). Kell went on to serve continuously in the role for more than twenty years. Ever present around intelligence matters for nearly half a century, Winston Churchill, as home secretary, oversaw Kell’s appointment. Adding a symmetry to Kell’s long career as spymaster, Churchill, when prime minister, would remove him from office in 1940.25 The Secret Service Bureau also had a foreign branch. Initially called MI1c, it soon restyled itself MI6, and was headed by the remarkable figure of Sir Mansfield Cumming, a retired naval officer. Despite losing his leg in a traffic accident in France, Cumming continued to run MI6, speeding down the corridors of Whitehall by planting his wooden leg firmly on a modified child’s scooter. MI5 was a small organisation and MI6 even smaller, but a key change had occurred. Hitherto, individual government departments, especially the India Office, had gathered intelligence locally and conducted espionage for their own purposes. Now Whitehall had an interdepartmental intelligence machine at the centre, delivering a ‘service’ to all government departments and anointed with the cult of specialness.26

After 1909, Asquith’s government went further. Harassed by continued public concern about nefarious German activities, it introduced the trappings of a secret state that previous successive governments had resisted for a century. Between 1909 and 1911, Asquith not only created the Secret Service Bureau, but also passed a draconian Official Secrets Act and allowed for much wider mail interception, something a Liberal government had banned as a diabolical infringement of liberties some fifty years before.27 The new Official Secrets Act, encouraged by Edmonds of MO(5), also helped the government to crack down on the press.28 Asquith, a facilitator rather than a dictatorial prime minister, also set up a new committee under Winston Churchill to consider how Britain would tackle ‘Aliens in Time of War’. With foreigners and exiles at the heart of this scare, the government created a register of aliens living in Britain. By 1913, it would have 11,000 Germans on this list, and had already prepared internment legislation.29 Notwithstanding domestic spy-fever, on the eve of the First World War much British intelligence activity lay elsewhere. Intrepid consuls, attachés and military officers carried out most of the spying on German armament programmes, while MI6 relied on the collection of specialist journals by travellers, or open observations, rather than actual espionage. Despite a lack of coordinated assessment, this material gradually filtered up into the higher reaches of government and influenced Britain’s diplomatic and strategic behaviour.30

The most immediate physical threat to Asquith came not from swarms of German spies, but from other quarters, notably the suffragette movement. In September 1909, two women were busy improving their shooting skills at a rifle range at 92 Tottenham Court Road with the rather improbable name of ‘Fairyland’. They intended to assassinate the prime minister, who was a firm opponent of votes for women. The two would-be assassins planned to join a group of suffragettes who had been picketing the gates of Parliament for eight weeks. Asquith passed them frequently, and therefore presented a tempting close-range target. Fortunately for the prime minister, the police received a tip-off. The informant was a Mrs Moore, a member of the Women’s Freedom League, but also a close friend of Asquith’s sister-in-law. She was an advocate of peaceful protest, and spent a great deal of time trying to dissuade her fellow suffragettes from violent acts.

Moore produced a letter from one of the two women who had been practising with a revolver.31 She refused to name names, but the police made enquiries at the shooting range, whose owner, Henry Morley, told them that two suffragettes had indeed been practising with a Browning automatic pistol. Their alarm was increased by the knowledge that the same range, and the same type of pistol, had been used only months earlier by an assassin who had killed Sir William Curzon-Wylie, aide to the secretary of state for India. Although undercover officers hung about for some days afterwards, neither woman returned. This left Asquith’s government with a dilemma familiar to later administrations dealing with terrorist threats. They knew full well that there would be serious recriminations if they did nothing and the prime minister was assassinated. The police did not remove the protesters from outside Parliament, but increased the number of officers there instead. They did not wish to give prominence to the idea of assassination, fearing that publicity might ‘act on the minds of these half-insane women, and might suggest effectively the commission of the very act which we wish to prevent’. Moreover, the removal of the pickets would be looked on by the women as an act of violence and injustice, and would ‘make them furious and more ready to commit such a crime’. In addition, the government thought that if there was an assassin, it would be easier to stop her if police knew she would be amongst the picketers, rather than walking ‘up and down between the House of Parliament and Downing Street at the hour when the P.M. may be expected to drive down’. Thanks to Mrs Moore, the prime minister remained able to pass in and out of Parliament unscathed.32 Recent research shows that the suffragettes included some notably dangerous groups, who fire-bombed churches and later sent a letter-bomb to Lloyd George. Some of the more violent women would go on to become active supporters of the extreme right, including Oswald Mosley’s blackshirts.33 Yet, remarkably, Asquith and all his successors through to the Second World War remained largely unprotected.

It was an act of assassination that finally triggered the outbreak of war in August 1914. Intelligence now had a more crucial – and real – role to play. The new secret services quickly pounced upon the small German espionage network in London, and rounded up all twenty-two known agents. The authorities had been assiduously following this spy ring, run from a barber shop in London by a naturalised German, since 1911. In an early intelligence success story, close monitoring had prevented the ring from passing useful information on to Berlin. Several hundred people were placed under surveillance. Predictably, the biggest problem was public paranoia. In the first two weeks of the war, the Metropolitan Police were forced to investigate thousands of people, with little to show for it. The newspapers followed this eagerly, and noted with satisfaction that by the end of November 1914 there had been over 100,000 reports of espionage, with 6,000 homes entered and searched.

Popular enthusiasm for war, combined with paranoia about spies, forced the Asquith government to look tough. Eleven German spies were shot at the Tower of London, a location chosen for its sinister appearance and reputation. More people were executed there during the First World War than under the Tudors. The amateurish German spies were not difficult to capture. They used lemon juice and peppermint oil instead of ink to render their reports to Berlin invisible, but were often arrested in possession of pen nibs corroded by the acidic lemon juice.34 Such tales allowed le Queux to continue to pump out his pulp-fiction spy thrillers. He maintained his phoney reputation as a spymaster to the end, and even now some of his relatives insist that he was assassinated by Russian agents in 1927.35

No one was safe from the paranoid public. Not even Charles Rennie Mackintosh, the Scottish architect and designer, who dared to sketch in the Suffolk village of Walberswick, close to the sea; or the composer Ralph Vaughan Williams, who sat down with a notebook to write ‘The Lark Ascending’ on cliffs near Margate. Both attracted vigorous police attention. Any Continental connection became a form of contagion. A Polish girl, who was a friend of the Asquith family, was fired from her teaching job simply out of fear that she might be mistaken for a German spy.36 The Asquith government stepped up internment, rounding up some 60,000 Germans living in Britain.37 Under popular pressure, and perhaps against his better judgement, the prime minister personally emphasised that all non-naturalised adult males ‘should, for their own safety, and that of the community, be segregated and interned, or, if over military age, repatriated’.38