{"id":60710,"date":"2015-07-06T12:36:47","date_gmt":"2015-07-06T11:36:47","guid":{"rendered":"https:\/\/www.transcend.org\/tms\/?p=60710"},"modified":"2015-07-06T12:46:09","modified_gmt":"2015-07-06T11:46:09","slug":"xkeyscore-part-ii-behind-the-curtain-a-look-at-the-inner-workings-of-nsa","status":"publish","type":"post","link":"https:\/\/www.transcend.org\/tms\/2015\/07\/xkeyscore-part-ii-behind-the-curtain-a-look-at-the-inner-workings-of-nsa\/","title":{"rendered":"XKEYSCORE (Part II) – Behind the Curtain: A Look at the Inner Workings of NSA"},"content":{"rendered":"

Second\u00a0in a series. Part 1 here<\/a>.<\/em><\/p>\n

2 Jul 2015 – <\/em>The sheer quantity of communications that XKEYSCORE processes, filters and queries is stunning. Around the world, when a person gets online to do anything \u2014 write an email, post to a social network, browse the web or play a video game \u2014 there\u2019s a decent chance that the Internet traffic her\u00a0device sends and receives is getting collected and processed by one of XKEYSCORE\u2019s hundreds of servers scattered across the globe.<\/p>\n

\"Illustration<\/a>

Illustration for The Intercept by Blue Delliquanti<\/p><\/div>\n

In order to make sense of such a massive and steady flow of information, analysts working for the National Security Agency, as well as partner spy agencies, have written thousands of snippets of code to detect different types of traffic and extract useful information from each type, according to documents dating up to 2013. For example, the system automatically detects if\u00a0a given piece of traffic is an email. If it is, the system\u00a0tags if it\u2019s from Yahoo or Gmail, if it contains an airline itinerary, if it\u2019s encrypted with PGP, or if the sender\u2019s language is set to Arabic, along with myriad other details.<\/p>\n

This global Internet surveillance network is powered by a somewhat clunky piece of software running on clusters of Linux servers. Analysts access XKEYSCORE\u2019s web interface to search its wealth of private information, similar to how ordinary people can search Google for public information.<\/p>\n

Based on documents provided by NSA whistleblower Edward Snowden, The Intercept<\/em> is shedding light on the inner workings of XKEYSCORE, one of the most extensive programs of mass surveillance in human history.<\/p>\n

How XKEYSCORE works under the hood<\/strong><\/p>\n

It is tempting to assume that expensive, proprietary operating systems and software must power XKEYSCORE, but it actually relies on an entirely open source stack. In fact, according to an analysis of an XKEYSCORE manual for new systems administrators from the end of 2012, the system may have design deficiencies that could leave it vulnerable to attack by an intelligence agency insider.<\/p>\n

XKEYSCORE is a piece of Linux software that is typically deployed on Red Hat servers. It uses the Apache web server and stores collected data in MySQL databases. File systems in a cluster are handled by the NFS distributed file system and the autofs service, and scheduled tasks are handled by the cron scheduling service. Systems administrators who maintain XKEYSCORE servers use SSH to connect to them, and they use tools such as rsync and vim, as well as a comprehensive command-line tool, to manage the software.<\/p>\n

John Adams, former security lead and senior operations engineer for Twitter, says that one of the most interesting things about XKEYSCORE\u2019s architecture is \u201cthat they were able to achieve so much success with such a poorly designed system. Data ingest, day-to-day operations, and searching is all poorly designed. There are many open source offerings that would function far better than this design with very little work. Their operations team must be extremely unhappy.\u201d<\/p>\n

Analysts connect to XKEYSCORE over HTTPS using standard web browsers such as Firefox. Internet Explorer is not supported. Analysts can log into the system with either a user ID and password or by using public key authentication.<\/p>\n

As of 2009, XKEYSCORE servers were located at more than 100 field sites all over the world. Each field site consists of a cluster of servers; the exact number differs depending on how much information is being collected at that site. Sites with relatively low traffic can get by with fewer servers, but sites that spy on larger amounts of traffic require more servers to filter and parse it all. XKEYSCORE has been engineered to scale in both processing power and storage by adding more servers to a cluster. According to a 2009 document, some field sites receive over 20 terrabytes of data per day. This is the equivalent of\u00a05.7 million songs, or over 13\u00a0thousand full-length films.<\/p>\n

\"This<\/a>

This map from a 2009 top-secret presentation does not show all of XKEYSCORE\u2019s field sites.<\/p><\/div>\n

When data is collected at an XKEYSCORE field site, it is processed locally and ultimately stored in MySQL databases at that site. XKEYSCORE supports a federated query system, which means that an analyst can conduct a single query from the central XKEYSCORE website, and it will communicate over the Internet to all of the field sites,\u00a0running\u00a0the query everywhere at once.<\/p>\n

There might be security issues with the XKEYSCORE system itself as well. As hard as software developers may try, it\u2019s nearly impossible to write bug-free source code. To compensate for this, developers often rely on multiple layers of security; if attackers can get through one layer, they may still be\u00a0thwarted by other layers. XKEYSCORE appears to do a bad job of this.<\/p>\n

When systems administrators log into XKEYSCORE servers to configure them, they appear to use a shared account, under the name \u201coper.\u201d Adams notes, \u201cThat means that changes made by an administrator cannot be logged.\u201d If one administrator does something malicious on an XKEYSCORE server using the \u201coper\u201d user, it\u2019s possible that the digital trail of what was done\u00a0wouldn\u2019t lead back to the administrator, since multiple operators use the account.<\/p>\n

There appears to be another way an ill-intentioned systems administrator may be able to cover their tracks. Analysts wishing to query XKEYSCORE sign in via a web browser, and their searches are logged. This creates an audit trail, on which the system relies to assure that users aren\u2019t doing overly broad searches that would pull up U.S. citizens\u2019 web traffic. Systems administrators, however, are able to run MySQL queries. The documents indicate that administrators have the ability to directly query the MySQL databases, where the collected data is stored, apparently bypassing the audit trail.<\/p>\n

AppIDs, fingerprints and microplugins<\/strong><\/p>\n

Collecting massive amounts of raw data is not very useful unless it is collated and organized in a way that can be searched. To deal with this problem, XKEYSCORE extracts and tags metadata and content from the raw data so that analysts can easily search it.<\/p>\n

This is done by using dictionaries of rules called appIDs, fingerprints and microplugins that are written in a custom programming language called GENESIS. Each of these can be identified by a unique name that resembles a directory tree, such as \u201cmail\/webmail\/gmail,\u201d \u201cchat\/yahoo,\u201d or \u201cbotnet\/blackenergybot\/command\/flood.\u201d<\/p>\n

One document detailing XKEYSCORE appIDs and fingerprints lists several revealing examples. Windows Update requests appear to fall under the \u201cupdate_service\/windows\u201d appID, and normal web requests fall under the \u201chttp\/get\u201d appID. XKEYSCORE can automatically detect Airblue travel itineraries with the \u201ctravel\/airblue\u201d fingerprint, and iPhone web browser traffic with the \u201cbrowser\/cellphone\/iphone\u201d fingerprint.<\/p>\n

PGP-encrypted messages are detected with the \u201cencryption\/pgp\/message\u201d fingerprint, and messages encrypted with Mojahedeen Secrets 2 (a type of encryption popular among supporters of al Qaeda) are detected with the \u201cencryption\/mojaheden2\u201d fingerprint.<\/p>\n

When new traffic flows into an XKEYSCORE cluster, the system tests the intercepted data against each of these rules and stores whether the traffic matches the pattern. A slideshow presentation from 2010 says that XKEYSCORE contains almost 10,000 appIDs and fingerprints.<\/p>\n

\"xks-app-for-that-540x405<\/a><\/p>\n

AppIDs are used to identify the protocol of traffic being intercepted, while fingerprints detect a specific type of content. Each intercepted stream of traffic gets assigned up to one appID and any number of fingerprints. You can think of appIDs as categories and fingerprints as tags.<\/p>\n

If multiple appIDs match a single stream of traffic, the appID with the lowest \u201clevel\u201d is selected\u00a0(appIDs with lower levels are more specific than appIDs with higher levels). For example, when XKEYSCORE is assessing a file attachment from Yahoo mail, all of the appIDs in the following slide will apply, however only \u201cmail\/webmail\/yahoo\/attachment\u201d will be associated with this stream of traffic.<\/p>\n

\"xks-appids-540x405<\/a><\/p>\n

To tie it all together, when an Arabic speaker logs into a\u00a0Yahoo email address, XKEYSCORE will store \u201cmail\/yahoo\/login\u201d as the associated appID. This stream of traffic will match the \u201cmail\/arabic\u201d fingerprint (denoting language settings), as well as the \u201cmail\/yahoo\/ymbm\u201d fingerprint (which detects Yahoo browser cookies).<\/p>\n

\"xks-appids-fingerprints-540x405<\/a><\/p>\n

Sometimes the GENESIS programming language, which largely relies on Boolean logic, regular expressions and a set of simple functions, isn\u2019t powerful enough to do the complex pattern-matching required to detect certain types of traffic. In these cases, as one slide puts it, \u201cPower users can drop in to C++ to express themselves.\u201d AppIDs or fingerprints that are written in C++ are called microplugins.<\/p>\n

Here\u2019s an example of a microplugin fingerprint for \u201cbotnet\/conficker_p2p_udp_data,\u201d which is tricky botnet traffic that can\u2019t be identified without complicated logic. A botnet is a collection of hacked computers, sometimes millions of them, that are controlled from a single point.<\/p>\n

\"xks-conficker-540x405<\/a><\/p>\n

Here\u2019s another microplugin that uses C++ to inspect intercepted Facebook chat messages and pull out details like the associated email address and body of the chat message.<\/p>\n

\"xks-facebook-chat-540x405<\/a><\/p>\n

One document from 2009 describes in detail four generations of appIDs and fingerprints, which begin\u00a0with only the ability to scan intercepted traffic for keywords, and end with the ability to write complex microplugins that can be deployed to field sites around the world in hours.<\/p>\n

If XKEYSCORE development has continued at a similar pace over\u00a0the last six years, it\u2019s likely considerably more powerful today.<\/p>\n

\u2014<\/p>\n

Documents published with this article:<\/em><\/p>\n