Venona project

The VENONA project was a long-running and highly secret collaboration between United States intelligence agencies and the United Kingdom's MI5 and GCHQ that involved the cryptanalysis of messages sent by several Soviet intelligence agencies. There were known to be at least 13 code words for this effort used by the US and UK. VENONA was the last code word for the project.

In the early years of the Cold War, VENONA would be an important source on Soviet intelligence activity for the Western powers. Although unknown to the public, it was a critical and guarded document behind many famous events of the early Cold War, such as the Rosenberg spying case.

Background

U.S. Army Signal Security Agency (commonly called Arlington Hall) codebreakers had intercepted large volumes of encrypted high-level Soviet diplomatic and intelligence traffic during and immediately after World War II. The British had stopped intercepting Soviet traffic, at Winston Churchill's orders, shortly after Germany invaded the Soviet Union in 1941, and had no traffic to contribute to the project after that time.

This traffic, some of which was encrypted with a one-time pad system, was stored and analyzed in relative secrecy by hundreds of cryptanalysts over a 40-year period starting in the early 1940s. Due to what turned out to be a serious blunder on the part of the Soviets - re-using pages of some of the one-time pads in other pads, which were then used for other messages - this traffic was vulnerable to cryptanalysis.

The Venona Project was initiated under orders from the Chief of Military Intelligence, Carter Clarke, who mistrusted Joseph Stalin. He feared that Stalin and Hitler would sign a peace treaty in order to focus Germany's military forces on the destruction of Great Britain and the U.S.

The break-in

The Soviet systems in general used a code to convert words and letters into numbers, to which an additive key (from one-time pads) were added, further disguising the content. Some brilliant cryptanalysis by American and British codebreakers revealed that some of the one-time pad material had incorrectly been reused by the Soviets (specifically, entire pages, although not complete books), which allowed decryption (sometimes only partial) of a small part of the traffic.

It was Arlington Hall's Lt. Richard Hallock, working on Soviet "Trade" traffic, who first discovered that the Soviets were re-using pages. Hallock and his colleagues (including Genevieve Feinstein, Cecil Phillips, Frank Lewis, Frank Wanat, and Lucille Campbell) went on to break into a significant amount of "Trade" traffic, recovering many one-time pad additive key tables in the process.

Meredith Gardner (left); most of the code breakers were young women.

A very young Meredith Gardner (of what would become the National Security Agency) then used this material to break in to what turned out to be NKVD (and later GRU) traffic, by reconstructing the code used to convert text to numbers. Samuel Chew and Cecil Phillips also made valuable contributions. On 20 December 1946, Gardner made the first break into the code, revealing the existence of Soviet espionage in the Manhattan Project. ^[1] Others worked in Washington in the State Department, Treasury, Office of Strategic Services, and even the White House. Very slowly, using assorted techniques ranging from traffic analysis to defector information, more of the messages were decrypted.

Claims have been made that information from physical theft of code books (a partially burned one was recovered by the Finns) to bugging embassy rooms in which text was entered into encrypting devices (analyzing the keystrokes by listening to them being punched in), contributed to achieving as much plaintext as was recovered. These latter claims are less than fully supported in the open literature.

One significant aid (mentioned by the NSA) in the early stages may have been work done in co-operation between the Japanese and Finnish cryptanalytic organizations; when the Americans broke into Japanese codes during WWII, they gained access to this information. There are also reports that copies of signals purloined from Soviet offices by the FBI were helpful in the cryptanalysis.

There has been speculation that the reason for the key material duplication was the increase in work (including key pad generation) in the period after the German attack in June of 1941. Other suggestions have it that it was Guderian's tanks just outside Moscow in early December that year which forced Moscow Centre to make such a fundamental error.

Results

The NSA reported that, according to the serial numbers of the Venona cables, thousands were sent, but only a fraction were available to the cryptanalysts. Approximately 2,200 of the messages were decrypted and translated; some 50 percent of the 1943 GRU-Naval Washington to Moscow messages were broken, but none for any other year, although several thousand were sent between 1941 and 1945. The decryption rate of the NKVD cables was:

1942 1.8%
1943 15.0%
1944 49.0%
1945 1.5%

Out of some hundreds of thousands of intercepted cyphertexts, it is claimed that under 3000 have been partially or wholly decrypted.

The Soviets eventually stopped reusing key pad material, possibly after learning of the US/British work from several of their agents, after which their secure traffic reverted to completely unreadable.

Public disclosure

Some of the earliest detailed public knowledge that Soviet code messages from WWII period had been broken came with the release of Robert Lamphere's book, The FBI-KGB War, in 1986. Lamphere had been the FBI liaison to the code-breaking activity, had considerable knowledge of the details of the "breakin," and was deeply involved in the counter-intelligence work that followed.

In 1995, a bipartisan Commission on Government Secrecy, with Senator Daniel Patrick Moynihan as chairman, was responsible for the release of Venona project materials (although many inside the NSA had also come to believe that the time had come to make it public, and argued internally for such a release).

Moynihan wrote "The Venona intercepts contained overwhelming proof of the activities of Soviet spy networks in America, complete with names, dates, places, and deeds." ^[2]

Significance

Main articles: Significance of Venona

The NSA followed Soviet intelligence traffic for only a few years in World War II, and decrypted only a small portion of that traffic. The Venona project was a thirty-eight year investigation conducted by the NSA and FBI counter-intelligence, and held classified for an additional fifteen years after the program ended. Researchers, historians, and the public continue to debate its significance and meaning. A few writers are skeptical of some claims made by certain analysts of the Venona documents, and urge a more cautious interpretive approach.^[3] This debate is discussed at Significance of Venona.

Document release issues

The release of VENONA translations involved careful consideration of the privacy interests of individuals mentioned, referenced, or identified in the translations. Some names have not been released when to do so would constitute an invasion of privacy. ^[4] (In at least one case, however, independent researchers have identified one of the subjects whose name was obscured.)

The NSA has failed to release all the VENONA documents as machine-readable text files. (Text processing technology could be used to extract information from the decrypts for historical research if the VENONA documents were released in this form.)

The NSA website states:

These historical documents are GIF images of formerly classified carbon paper and reports that have been declassified. Due to the age and poor quality of some of the GIF images, a screen reader may not be able to process the images into word documents." [...] "individuals may request that the government provide auxiliary aids or services to ensure effective communication of the substance of the documents. For such requests, please contact the Public Affairs Office at 301-688-6524.

Notes

^1 Moynihan Commssion on Government Secrecy, Appendix A, The Experience of The Bomb (1997)
^2 Secrecy: The American Experience; by Daniel Patrick Moynihan, New Haven: Yale University Press 1998, pg. 15.
^3 Ellen Schrecker, Many are the Crimes: McCarthyism in America (Boston: Little Brown, 1998) pp. xvii-xviii
^4 VENONA Historical Monograph #4 National Security Agency Archives, Cryptological Museum

References

NSA official VENONA site
Moynihan Commssion Report on Government Secrecy (1997)
Selected Venona Messages
MI5 Releases to the National Archives
Venona Chronology 1939-1996 per Denis Naranjo
Interview with Cecil Philips

Additional background material

Daniel Patrick Moynihan, Secrecy: The American Experience (New Haven: Yale University Press 1998) ISBN 0300080794
Richard J. Aldrich, The Hidden Hand: Britain, America, and Cold War Secret Intelligence (New York: Overlook Press, 2002) ISBN 1585672742.
James Bamford, Body of Secrets: Anatomy of the Ultra-Secret National Security Agency (Anchor Books) ISBN 0385499086. (See also the same author's earlier, The Puzzle Palace, also about the NSA.)
Steven Budiansky, Battle of Wits - An overview in one volume of cryptography in WWII.

^ breakin
^ Secrecy15
^ Schrecker
^ Monograph4

[1] reakin

[2] Secrecy15

[3] Schrecker

[4] Monograph4

[1]

[2]

[3]

[4]