How Echelon works..
As some of you know, I used to work in Counter-Intelligence and have a general interest in Spying, Intelligence gathering, and anti-Terrorism. I wrote this article to explain where I see the current state of corporate security and monitoring moving towards.
Please take with appropriate salt.
How Echelon works, by 1Whiterabbit.
"ECHELON is a highly secretive world-wide signals intelligence and analysis network run by the UKUSA (English speaking) community. ECHELON can capture radio and satellite communications, telephone calls, faxes and emails nearly anywhere in the world and includes computer automated analysis and sorting of intercepts. ECHELON is estimated to intercept up to 3 billion communications every day... Reportedly created to monitor the military and diplomatic communications of the Soviet Union and its East Bloc allies, ECHELON is today believed to also search for hints of terrorist plots, drug-dealers' plans, and political and diplomatic intelligence. But some critics claim the system is also being used for large-scale commercial theft and invasion of privacy."
From: http://en.wikipedia.org/wiki/ECHELON
This is document is all theory and no classified documents are at risk here, however with a background in DoE and DoD and knowledge of systems currently available I feel that this is a reasonable guess into the capabilities of any agency.
This is how a corporation (government or otherwise) could most effectively create a Echelon worldwide monitoring system with a minimal amount of data overload for the limited (100+) personnel that would be monitoring an entire world of constantly flowing information.
Granted, any agency would have to have the right hooks into telephone systems, encryption keys, Operating Systems, data flows, radio and television, etc..
A large corporation could do this internally with a surprisingly minimal amount of effort using reduced personnel and custom built software, a government would obviously need a much more elaborate system.
Step one. Create an index all information using taps, hooks, and spiders. "An index is a feature in a database that allows quick access to the rows in a table. The index is created using one or more columns of the table. Not only is the index smaller than the original table (due to having fewer columns), but it is optimized for quick searching, usually via a balanced tree. "http://en.wikipedia.org/wiki/Index_%28database%29
This is how Google archives WebPages: http://www.googleguide.com/google_works.html
"Matt Cutts, a key Google engineer, used to work for the National Security Agency. Google wants to hire more people with security clearances, so that they can peddle their corporate assets to the spooks in Washington." http://www.google-watch.org/bigbro.html
*Multiple languages need to be converted to the readers language with a key marking the origination (but it would be simpler to store it and have multiple language personnel reading it in the original format.)
*Telephone conversations need to be stored in two formats. Voice needs to be changed into text in order to be searchable as well as stored as a graphical wave snapshot in order to compare voice print's to specific individuals for tracking and comparison. Advanced compression can be used to save space without data quality loss.
This graph is like a fingerprint for voice print and would enable a search of supposedly anonymous conversations to be identified based on past emails once the fingerprint is identified.
*Email and print (newspapers, scans of collected mail) needs to be down converted into text (from Rich Text or handwriting) for space concerns.
*Encrypted email and communication needs to be stored in a separate database for further processing using hash keys (see below).
----------Handling Encrypted transmissions----------
Encryption works by converting a word into a hash using a known key, the system then compares the hash (without storing the password key) for access.
It works something like this.
password + complex algorithm = Hash key
Here is a simple example:
pass + 1234 = p1a2s3s4
So the hash key is p1a2s3s4. The system stores p1a2s3s4 but doesn't know the original password (pass) just the algorithm (____ + 1234), and can not get the password without going through multiple combinations of attempts, like so.
a+1234
aa+1234
aaa+1234
and so on..
This is how brute force password cracking works, and it takes a long time on a single system. Which is why you change your password every 30-40 days.
But there is a simpler way if you have the storage space and processing power.
Store all the hash keys and compare these in order to compare against an encrypted transmission.
So when the encrypted transmission comes in with a transmitted hash key of p1a2s3s4, the Echelon system simply runs through the hash set that it has stored and extracts the key and known password that matches.
This takes a little time, but not too much if you break up the process and split the message across a hundred systems using parallel processing.
----------------------------------------------------
The final step is to create a viewer for new (and old) information that your employees will monitor.
"The RSS formats provide web content or summaries of web content together with links to the full versions of the content, and other meta-data. This information is delivered as an XML file called an RSS feed, web feed, RSS stream, or RSS channel. In addition to facilitating syndication, RSS allows a website's frequent readers to track updates on the site using an aggregator." http://en.wikipedia.org/wiki/RSS_%28file_format%29
A webpage can be created parsing the information out to personnel that have individual content.
The information is stored in a "Simple - Summary - Larger Summary - Complete Page" format.
Information can be parsed into categories such as "Main Suspects - Terrorist actions - Bombs - ___" etc..
An excellent example is the website http://www.180n.com/
The summary can be expanded and then a simple vote button highlights information for additional research or scrutiny.
There are other methods of parsing the data, "cloud view" highlights important conversations while still showing additional information.
This is from: http://digg.com/diggall/cloud
Personnel can be split into groups or a hierarchy for data triage. You have general watchers, new and current data watchers, and key suspect watchers.
But "who will watch the watchmen?" The original question, first posed in Latin, is as critical now as it was 2,000 years ago. Absolute power has to be monitored and kept in check. An agency that has the power to protect us from enemies also has the power to do it's citizens great harm in the process.
So for any government agencies I pose this warning:
"Battle not with monsters, lest you become a monster, and if you gaze into the abyss, the abyss also gazes into you." -Fredrich Nietzsche.
Finally, despite what they are saying today, this has been going on for decades:
Bush says Surveillance program protects country:
http://www.usatoday.com/news/washington/2006-01-23-bush_x.htm
Campaign To Justify Spying Intensifies:
http://www.washingtonpost.com/wp-dyn/content/article/2006/01/23/AR2006012300754.html
Wiretaps Are a Winner for W:
http://frontpagemagazine.com/Articles/ReadArticle.asp?ID=21009
Spying: The dark side of technology:
http://www.napavalleyregister.com/articles/2006/01/24/business/local/iq_3267517.txt
Bush opens PR blitz on warrantless [sic] wiretaps, Intelligence official says program is not 'mining' data:
http://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2006/01/24/MNGQ4GRUHP1.DTL
PS: Spooks, please feel free to contact me at your discretion.
Please take with appropriate salt.
How Echelon works, by 1Whiterabbit.
"ECHELON is a highly secretive world-wide signals intelligence and analysis network run by the UKUSA (English speaking) community. ECHELON can capture radio and satellite communications, telephone calls, faxes and emails nearly anywhere in the world and includes computer automated analysis and sorting of intercepts. ECHELON is estimated to intercept up to 3 billion communications every day... Reportedly created to monitor the military and diplomatic communications of the Soviet Union and its East Bloc allies, ECHELON is today believed to also search for hints of terrorist plots, drug-dealers' plans, and political and diplomatic intelligence. But some critics claim the system is also being used for large-scale commercial theft and invasion of privacy."
From: http://en.wikipedia.org/wiki/ECHELON
This is document is all theory and no classified documents are at risk here, however with a background in DoE and DoD and knowledge of systems currently available I feel that this is a reasonable guess into the capabilities of any agency.
This is how a corporation (government or otherwise) could most effectively create a Echelon worldwide monitoring system with a minimal amount of data overload for the limited (100+) personnel that would be monitoring an entire world of constantly flowing information.
Granted, any agency would have to have the right hooks into telephone systems, encryption keys, Operating Systems, data flows, radio and television, etc..
A large corporation could do this internally with a surprisingly minimal amount of effort using reduced personnel and custom built software, a government would obviously need a much more elaborate system.
Step one. Create an index all information using taps, hooks, and spiders. "An index is a feature in a database that allows quick access to the rows in a table. The index is created using one or more columns of the table. Not only is the index smaller than the original table (due to having fewer columns), but it is optimized for quick searching, usually via a balanced tree. "http://en.wikipedia.org/wiki/Index_%28database%29
This is how Google archives WebPages: http://www.googleguide.com/google_works.html
"Matt Cutts, a key Google engineer, used to work for the National Security Agency. Google wants to hire more people with security clearances, so that they can peddle their corporate assets to the spooks in Washington." http://www.google-watch.org/bigbro.html
*Multiple languages need to be converted to the readers language with a key marking the origination (but it would be simpler to store it and have multiple language personnel reading it in the original format.)
*Telephone conversations need to be stored in two formats. Voice needs to be changed into text in order to be searchable as well as stored as a graphical wave snapshot in order to compare voice print's to specific individuals for tracking and comparison. Advanced compression can be used to save space without data quality loss.
This graph is like a fingerprint for voice print and would enable a search of supposedly anonymous conversations to be identified based on past emails once the fingerprint is identified.
*Email and print (newspapers, scans of collected mail) needs to be down converted into text (from Rich Text or handwriting) for space concerns.
*Encrypted email and communication needs to be stored in a separate database for further processing using hash keys (see below).
----------Handling Encrypted transmissions----------
Encryption works by converting a word into a hash using a known key, the system then compares the hash (without storing the password key) for access.
It works something like this.
password + complex algorithm = Hash key
Here is a simple example:
pass + 1234 = p1a2s3s4
So the hash key is p1a2s3s4. The system stores p1a2s3s4 but doesn't know the original password (pass) just the algorithm (____ + 1234), and can not get the password without going through multiple combinations of attempts, like so.
a+1234
aa+1234
aaa+1234
and so on..
This is how brute force password cracking works, and it takes a long time on a single system. Which is why you change your password every 30-40 days.
But there is a simpler way if you have the storage space and processing power.
Store all the hash keys and compare these in order to compare against an encrypted transmission.
So when the encrypted transmission comes in with a transmitted hash key of p1a2s3s4, the Echelon system simply runs through the hash set that it has stored and extracts the key and known password that matches.
This takes a little time, but not too much if you break up the process and split the message across a hundred systems using parallel processing.
----------------------------------------------------
The final step is to create a viewer for new (and old) information that your employees will monitor.
"The RSS formats provide web content or summaries of web content together with links to the full versions of the content, and other meta-data. This information is delivered as an XML file called an RSS feed, web feed, RSS stream, or RSS channel. In addition to facilitating syndication, RSS allows a website's frequent readers to track updates on the site using an aggregator." http://en.wikipedia.org/wiki/RSS_%28file_format%29
A webpage can be created parsing the information out to personnel that have individual content.
The information is stored in a "Simple - Summary - Larger Summary - Complete Page" format.
Information can be parsed into categories such as "Main Suspects - Terrorist actions - Bombs - ___" etc..
An excellent example is the website http://www.180n.com/
The summary can be expanded and then a simple vote button highlights information for additional research or scrutiny.
There are other methods of parsing the data, "cloud view" highlights important conversations while still showing additional information.
This is from: http://digg.com/diggall/cloud
Personnel can be split into groups or a hierarchy for data triage. You have general watchers, new and current data watchers, and key suspect watchers.
But "who will watch the watchmen?" The original question, first posed in Latin, is as critical now as it was 2,000 years ago. Absolute power has to be monitored and kept in check. An agency that has the power to protect us from enemies also has the power to do it's citizens great harm in the process.
So for any government agencies I pose this warning:
"Battle not with monsters, lest you become a monster, and if you gaze into the abyss, the abyss also gazes into you." -Fredrich Nietzsche.
Finally, despite what they are saying today, this has been going on for decades:
Bush says Surveillance program protects country:
http://www.usatoday.com/news/washington/2006-01-23-bush_x.htm
Campaign To Justify Spying Intensifies:
http://www.washingtonpost.com/wp-dyn/content/article/2006/01/23/AR2006012300754.html
Wiretaps Are a Winner for W:
http://frontpagemagazine.com/Articles/ReadArticle.asp?ID=21009
Spying: The dark side of technology:
http://www.napavalleyregister.com/articles/2006/01/24/business/local/iq_3267517.txt
Bush opens PR blitz on warrantless [sic] wiretaps, Intelligence official says program is not 'mining' data:
http://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2006/01/24/MNGQ4GRUHP1.DTL
PS: Spooks, please feel free to contact me at your discretion.