Jump to content

Open-source intelligence: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
GreenC bot (talk | contribs)
Bots
2,547,812 edits
Rescued 1 archive link. Wayback Medic 2.5
Tag: Reverted
Line 200: Line 200:
* [https://www.academia.edu/20836279/The_Social_Media_OSINT_Challenge_to_US_Intelligence_Culture_not_Gigabytes Abdelrahman Rashdan: 'The Social Media OSINT Challenge to US Intelligence: Culture Not Gigabytes', in New Media Politics Rethinking Activism and National Security in Cyberspace. Ed. Banu Baybars-Hawks. Cambridge Scholars Publishing, 2015]
* [https://www.academia.edu/20836279/The_Social_Media_OSINT_Challenge_to_US_Intelligence_Culture_not_Gigabytes Abdelrahman Rashdan: 'The Social Media OSINT Challenge to US Intelligence: Culture Not Gigabytes', in New Media Politics Rethinking Activism and National Security in Cyberspace. Ed. Banu Baybars-Hawks. Cambridge Scholars Publishing, 2015]
* {{Cite book|last=Hassan, Nihad A.|url=https://www.worldcat.org/oclc/1043830928|title=Open source intelligence methods and tools : a practical guide to online intelligence|others=Hijazi, Rami|date=30 June 2018|isbn=978-1-4842-3213-2|location=[United States]|oclc=1043830928}}
* {{Cite book|last=Hassan, Nihad A.|url=https://www.worldcat.org/oclc/1043830928|title=Open source intelligence methods and tools : a practical guide to online intelligence|others=Hijazi, Rami|date=30 June 2018|isbn=978-1-4842-3213-2|location=[United States]|oclc=1043830928}}
;In Popular culture
* film https://en.wikipedia.org/wiki/Three_Days_of_the_Condor


== External links ==
== External links ==

Revision as of 00:20, 28 June 2022

Open-source intelligence (OSINT) is the collection and analysis of data gathered from open sources (overt and publicly available sources) to produce actionable intelligence. OSINT is primarily used in national security, law enforcement, and business intelligence functions and is of value to analysts who use non-sensitive intelligence in answering classified, unclassified, or proprietary intelligence requirements across the previous intelligence disciplines.[1]

OSINT sources can be divided up into six different categories of information flow:[2]

OSINT is distinguished from research in that it applies the process of intelligence to create tailored knowledge supportive of a specific decision by a specific individual or group.[3]

Definition

OSINT is defined by both the U.S. Director of National Intelligence and the U.S. Department of Defense (DoD), as intelligence "produced from publicly available information that is collected, exploited, and disseminated in a timely manner to an appropriate audience for the purpose of addressing a specific intelligence requirement."[4] As defined by NATO, OSINT is intelligence "derived from publicly available information, as well as other unclassified information that has limited public distribution or access."[5]

According to political scientist Jeffrey T. Richelson, “open source acquisition involves procuring verbal, written, or electronically transmitted material that can be obtained legally. In addition to documents and videos available via the Internet or provided by a human source, others are obtained after U.S. or allied forces have taken control of a facility or site formerly operated by a foreign government or terrorist group.”[6]

Security researcher Mark M. Lowenthal defines OSINT as “any and all information that can be derived from overt collection: all types of media, government reports and other documents, scientific research and reports, commercial vendors of information, the Internet, and so on. The main qualifiers to open-source information are that it does not require any type of clandestine collection techniques to obtain it and that it must be obtained through means that entirely meet the copyright and commercial requirements of the vendors where applicable."[7]

History

Seal of the 9/11 Commission

OSINT in the United States traces its origins to the creation of the Foreign Broadcast Monitoring Service (FBMS), an agency responsible for the monitoring of foreign broadcasts. An example of their work is reflected in the application of the correlation of changes in the price of oranges in Paris with that of railway bridges being bombed successfully.[8]

The Aspin-Brown Commission stated in 1996 that US access to open sources was "severely deficient" and that this should be a "top priority" for both funding and DCI attention.[9]

In July 2004, following the September 11 attacks, the 9/11 Commission recommended the creation of an open-source intelligence agency.[10] In March 2005, the Iraq Intelligence Commission recommended[11] the creation of an open-source directorate at the CIA.

Following these recommendations, in November 2005 the Director of National Intelligence announced the creation of the DNI Open Source Center. The Center was established to collect information available from "the Internet, databases, press, radio, television, video, geospatial data, photos and commercial imagery."[12] In addition to collecting openly available information, it would train analysts to make better use of this information. The center absorbed the CIA's previously existing Foreign Broadcast Information Service (FBIS), originally established in 1941, with FBIS head Douglas Naquin named as director of the center.[13] Then, following the events of 9/11 the Intelligence Reform and Terrorism Prevention Act merged FBIS and other research elements into the Office of the Director of National Intelligence creating the Open Source Enterprise.

Furthermore, the private sector has invested in tools which aid in OSINT collection and analysis. Specifically, In-Q-Tel, a Central Intelligence Agency supported venture capital firm in Arlington, VA assisted companies develop web-monitoring and predictive analysis tools.

In December 2005, the Director of National Intelligence appointed Eliot A. Jardines as the Assistant Deputy Director of National Intelligence for Open Source to serve as the Intelligence Community's senior intelligence officer for open source and to provide strategy, guidance and oversight for the National Open Source Enterprise.[14] Mr. Jardines has established the National Open Source Enterprise[15] and authored intelligence community directive 301. In 2008, Mr. Jardines returned to the private sector and was succeeded by Dan Butler who is ADDNI/OS[16] and previously Mr. Jardines' Senior Advisor for Policy.[17]

Value

OSINT is valuable because it has less rigorous processing and exploitation processes and timelines than more technical intelligence disciplines such as HUMINT, SIGINT, MASINT, GEOINT, etc. Additionally, OSINT collects a valuable variety of opinions because it encompasses a great variety of sources.

According to the Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction report submitted in March 2005, OSINT must be included in the all-source intelligence process for the following reasons (as stated in the report):

  1. The ever-shifting nature of intelligence needs compels the IC (Intelligence Community) to quickly and easily understand a wide range of foreign countries and cultures. – … today's threats are rapidly changing and geographically diffuse; an intelligence analyst may be forced to shift rapidly from one topic to the next. Increasingly, IC professionals need to quickly assimilate social, economic, and cultural information about a country—information often detailed in open sources.
  2. Open-source information provides a base for understanding classified materials. Despite large quantities of classified material produced by the IC, the amount of classified information produced on any one topic can be quite limited, and may be taken out of context if viewed only from a classified-source perspective. A notable example relates to terrorism, where open-source information can fill gaps and create links that allow analysts to better understand fragmented intelligence, rumored terrorist plans, possible means of attack, and potential targets.
  3. Open-source materials can protect sources and methods. Sometimes an intelligence judgment that is actually informed with sensitive, classified information can be defended on the basis of open-source reporting. This can prove useful when policy-makers need to explain policy decisions or communicate with foreign officials without compromising classified sources.
  4. Only open source can store history. A robust open-source program can, in effect, gather data to monitor the world's cultures and how they change with time. This is difficult, if not impossible, using the snapshots provided by classified collection methods.[18]

Process

OSINT is a highly diverse form of intelligence collection and analysis. It does not have its own agency, however, units are scattered within the Department of Defense and the State Department.[19] Most OSINT collectors need to take precautions while collecting information from the Internet. This can come in the form of using a VPN to anonymize their identity and collect information more discreetly. This is where evaluating sources becomes important to the overall OSINT collection and analysis process. An OSINT analyst needs intelligence evaluation to determine a true process or expose a false process that would affect predicting the future. Finally, the analysts need to find use of the evaluated intelligence so that it can be incorporated into a finished classified, unclassified, or proprietary intelligence product.

See also: Big Data

Information collection in OSINT is generally a different problem from collection in other intelligence disciplines where obtaining the raw information to be analyzed may be the major difficulty, particularly if it is to be obtained from non-cooperative targets. In OSINT, the chief difficulty is in identifying relevant, reliable sources from the vast amount of publicly available information.[20] However, this is not as great a challenge for those who know how to access local knowledge and how to leverage human experts who can create new tailored knowledge on the fly. [citation needed]

Open Source Intelligence analysis software

There are several categories of tools intended for OSINT analysis. The first category includes open source tools to query multiple search engines simultaneously such as federated search engines and both proprietary and open source software projects which exploit the APIs of both free and paywalled open source and publicly available information (PIA) sources. This category also includes social media search engines and search engines of domains and people such as data aggregator services (sometimes referred to as data brokers or information brokers). The ecosystem of bespoke OSI gathering, collection and analysis tools includes specialized functions related to the following (partial) list of data types and categories as illustrated in the open-source project, OSINT Framework:[21]

  • Username
  • Email Address
  • Domain Name
  • IP Address
  • Images
  • Videos
  • Documents
  • Instant messaging
  • Telephone Numbers
  • Public Records
  • Business Records
  • Transportation
  • Geolocation Tools / Maps
  • Forums / Blogs / IRC
  • Archives
  • Metadata
  • Dark Web
  • Digital Currency


The second category is designed for big data analytics platforms such as Palantir, DataWalk, i2 as well as node-based graph tools such as Maltego which combine OSINT insight with local, internal data for further visual analysis and to conduct link analysis to identify connections across a large volume of records.

Tools

The web browser is a powerful OSINT tool that provides access to numerous websites and both open source and proprietary software tools that are either purpose-built for open source information collection or which can be exploited for the purposes of either gathering of open source information or to facilitate analysis and validation to provide intelligence. A cottage industry of both for-profit and not-for-profit investigative and educational groups such as Bellingcat, IntelTechniques SANS and others offer indices, books, podcasts and video training materials on OSINT tools and techniques. Books such as Michael Bazzell's Open Source Intelligence Techniques serve as indices to resources across mulitple domains but according the author, due to the rapidly changing information landscape, some tools and techniques change or become obsolete frequently, hence it is imperative for OSINT researchers to study, train and survey the landscape of source material regularly.[22] A guide by Ryan Fedasiuk, an analyst at the Center for Security and Emerging Technology, lists six tools open-source analysts can use to stay safe and utlize operational security (OPSEC) when conducting online investigations. These include VPNs, cached webpages, digital archive services, URL and file scanners, browser sandbox applications, and antivirus software.[23]

Numerous lists of aggregated OSINT content are available on the web. The OSINT Framework contains over 30 primary categories of tools and is maintained as an open source project on GitHub.[24] A sample list of some tools used for collecting open source intelligence:

OSINT community disciplines

United States

Government

There are a large number of open-source activities taking place throughout the US Government. Frequently, these open-source activities are described as "media monitoring", "media analysis", "internet research" and "public surveys" but are open source nonetheless.

The Library of Congress sponsors the Federal Research Division (FRD) which conducts a great deal of tailored open-source research on a fee-for-service basis for the executive branch.

Intelligence

The US Intelligence Community's open-source activities (known as the National Open Source Enterprise) are dictated by Intelligence Community Directive 301 promulgated by the Director of National Intelligence.[25] The Directive establishes the authorities and responsibilities of the Assistant Deputy Director of National Intelligence for Open Source (ADDNI/OS), the DNI's Open Source Center and the National Open Source Committee.

Prior to the establishment of the National Open Source Enterprise, the Foreign Broadcast Information Service (FBIS), established in 1941, was the government's primary open-source unit, transcribing and translating foreign broadcasts. It absorbed the Defense Department's Joint Publications Research Service (JPRS), which did a similar function with foreign printed materials, including newspapers, magazines, and technical journals.

Armed forces

The former Under-Secretary of Defense for Intelligence, Dr. Stephen Cambone encouraged in part by the Defense Science Board reports on strategic communication and transition to and from hostilities, created the Defense Open Source Program (DOSP). The current under-secretary of defense for intelligence is assigned executive agency for this program to the Defense Intelligence Agency.

U.S. military offices that engage in OSINT activities include:

Homeland Security

The Department of Homeland Security has an active open-source intelligence unit. In congressional testimony before the House Homeland Security Committee's Intelligence, Information Sharing and Terrorism Risk Assessment Subcommittee, Undersecretary of Homeland Security Charles Allen indicated on February 14, 2007, that he had established the "Domestic Open Source Enterprise" to support the Department's OSINT needs and that of state, local,Ο and tribal partners.

Law enforcement

The law enforcement OSINT community applies open-source intelligence (OSINT) to the prediction, prevention, investigation, and prosecution of criminals including terrorists. Additionally, fusion centers around the US are increasingly utilizing OSINT to support their intelligence generation and investigations.

Examples of successful law enforcement OSINT include Scotland Yard OSINT; Royal Canadian Mounted Police (RCMP) OSINT.

INTERPOL and EUROPOL experimented with OSINT units for a time, but they appear to have atrophied with the departure of their individual champions.

New York Police Department (NYPD) is known to have an OSINT unit, as does the Los Angeles County Sheriff's Department, housed within the Emergency Operations Bureau and affiliated with the LA Joint Regional Intelligence Center.

Business

Business OSINT encompasses Commercial Intelligence, Competitor Intelligence, and Business Intelligence, and is often a chief area of practice of private intelligence agencies.

Businesses may use information brokers and private investigators to collect and analyze relevant information for business purposes which may include the media, deep web, web 2.0 and commercial content.

Private specialized business

Another related business group within the United States that relies upon OSINT is the commercial bail-bond industry. This related industry, servicing the court system, is apart from the above Business Intelligence sector. OSINT is useful to bail-bond agencies that employ a private fugitive recovery agency to locate and apprehend their absent client; i.e., a criminal defendant who has failed to appear for court and subsequently a warrant for arrest was issued. OSINT is the first method of discovery to help locate the defendant when initial interviewing of the bond co-signers, defendant's relatives and friends is lacking. OSINT gathering leads the investigator to discover an alternate hypothesis to analyze and then match relevant data for making a prediction regarding the fugitive's location; e.g., data is scrubbed from web access on Facebook entries, Twitter messages, and Snapchat.

Should those methods fail, the next step is to seek the specialized behavioral intelligence services that reference OSINT to aid in establishing the veracity of subjects during the forensic interview and is used to create a behavioral profile. OSINT data is correlated with interview data to include a variety behavioral patterns; e.g., a list of daily personal contacts, habits of activities, visited places of interest, vehicles used, favorite group involvements, etc. According to the director, psychologist and forensic interviewer at MN-Behavioral Intelligence Agency, (2016) OSINT data base has to be critically filtered and analyzed before it can be applied within investigative interviewing and interrogation.

Risks for practitioners

A main hindrance to practical OSINT is the volume of information it has to deal with ("information explosion"). The amount of data being distributed increases at a rate that it becomes difficult to evaluate sources in intelligence analysis. To a small degree the work has sometimes been done by amateur crowd-sourcing.[26]

Accredited journalists have some protection in asking questions, and researching for recognized media outlets. Even so, they can be imprisoned, even executed, for seeking out OSINT. Private individuals illegally collecting data for a foreign military or intelligence agency is considered espionage in most countries. Of course, espionage that is not treason (i.e. betraying one's country of citizenship) has been a tool of statecraft since ancient times.[27]

See also

References

  1. ^ Schwartz, Leo (March 7, 2022). "Amateur open-source researchers went viral unpacking the war in Ukraine". Rest of World. Retrieved 8 March 2022.
  2. ^ Richelson, Jeffrey (2016). The US Intelligence Community. ISBN 978-0813349183.
  3. ^ "Spy Agencies Turn to Newspapers, NPR, and Wikipedia for Information: The intelligence community is learning to value 'open-source' information". Archived from the original on 2012-10-23. Retrieved 2008-09-15.
  4. ^ "As defined in Sec. 931 of Public Law 109-163, entitled, "National Defense Authorization Act for Fiscal Year 2006."". Archived from the original on 2008-11-12. Retrieved 2006-12-08.
  5. ^ "NATOTermOTAN". nso.nato.int. Retrieved 2021-04-02.
  6. ^ Richelson, Jeffrey T (2015-07-14). The U.S. Intelligence Community. Avalon Publishing. ISBN 9780813349190. Retrieved 15 May 2017.
  7. ^ Lowenthal, Mark M. (2005), "Open-Source Intelligence: New Myths, New Realities", in George, Roger Z; Kline, Robert D (eds.), Intelligence and the national security strategist : enduring issues and challenges, Lanham: Rowman and Littlefield, ISBN 9780742540392
  8. ^ Bornn, D Marshall (9 Jan 2013). "Service members, civilians learn to harness power of 'Open Source' information". www.army.mil. Archived from the original on 9 December 2017. Retrieved 14 May 2017.
  9. ^ Lowenthal, Mark; Clark, Robert (2015). The Five Disciplines of Intelligence Collection. CQ Press. p. 18. ISBN 978-1483381114.
  10. ^ See page 413 of the 9-11 Commission Report (pdf) Archived 2007-07-05 at the Wayback Machine.
  11. ^ McLaughlin, Michael (June 2012). "Using open source intelligence for cybersecurity intelligence". ComputerWeekly.com. Archived from the original on 2018-06-29. Retrieved 2018-06-29.
  12. ^ Office of the Director of National Intelligence. "ODNI Announces Establishment of Open Source Center Archived 2006-06-23 at the Wayback Machine". Press release, 8 November 2005.
  13. ^ Ensor, David. "The Situation Report: Open source intelligence center Archived 2007-03-25 at the Wayback Machine". CNN, 8 November 2005.
  14. ^ Office of the Director of National Intelligence "ODNI Senior Leadership Announcement Archived 2006-06-23 at the Wayback Machine". Press release, 7 December 2005.
  15. ^ "National Open Source Entreprise Vision Statement" Archived 2007-09-28 at the Wayback Machine May 2006
  16. ^ DNI Open Source Conference 2008 "Decision Advantage" agenda, Office of the Director of National Intelligence, July 2008. Archived 2010-04-17 at the Wayback Machine
  17. ^ DNI Open Source Conference 2007 "Expanding the Horizons" agenda, Office of the Director of National Intelligence, July 2007. Archived 2008-08-01 at the Wayback Machine
  18. ^ (The Commission on the Intelligence Capabilities, ). Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction
  19. ^ "Reexamining the Distinction Between Open Information and Secrets – Central Intelligence Agency". www.cia.gov. Archived from the original on 2018-06-08. Retrieved 2018-06-29.
  20. ^ Hudnall, Ken (2011). "Intelligence Failures". No Safe Haven: Homeland Insecurity. Grave Distractions Publications. ISBN 9781452493923.
  21. ^ OSINT Framework, lockFALE, 2022-05-02, retrieved 2022-05-02
  22. ^ "Books by Michael Bazzell". inteltechniques.com. Retrieved 2022-05-02.
  23. ^ Fedasiuk, Ryan (2022-04-06). "Into the Jungle: Best Practices for Open-Source Researchers". Center for Security and Emerging Technology. Retrieved 2022-04-22.
  24. ^ OSINT Framework, lockFALE, 2022-05-02, retrieved 2022-05-02
  25. ^ DNI Intelligence Community Directive 301 – "National Open Source Enterprise" Archived 2015-12-22 at the Wayback Machine 11 July 2006.
  26. ^ "Bellingcat's Eliot Higgins Explains Why Ukraine Is Winning the Information War". Time. Retrieved 2022-04-13.
  27. ^ Sun Tzu (Warring States period), The Art of War, Chapter 13: "Hostile armies may face each other for years, striving for the victory which is decided in a single day. This being so, to remain in ignorance of the enemy's condition simply because one grudges the outlay of 2 hundred ounces of silver in honors and emoluments, is the height of inhumanity."
  • WashTimes.com, Washington Times – CIA mines 'rich' content from blogs, 19 April 2006
  • GCN.com, Government Computer News – Intelligence units mine the benefits of public sources 20 March 2006
  • AFCEA.org, SIGNAL Magazine – Intelligence Center Mines Open Sources March 2006
  • FindAcricles.com, Military Intelligence Professional Bulletin October–December, 2005 by Barbara G. Fast
  • FAS.org, Congressional Testimony on OSINT and Homeland Security 21 June 2005
  • FirstMonday.org, Open Source Intelligence by Stalder and Hirsh, 15 May 2002
  • Forbes.com, When Everyone Can Mine Your Data by Taylor Buley, 11.21.08]
  • Thompson, Clive (2006-12-03). "Open-Source Spying". The New York Times. Retrieved 2018-05-29.
  • Cnet.com, Maltego and the science of 'open-source' snooping by Matt Asay, November 25, 2008

Literature

Scientific Publications
In Popular culture
Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=Open-source_intelligence&oldid=1095371474"