On Fri, Jun 7, 2013 at 6:47 AM, Eugen Leitl <eu...@leitl.org> wrote: > but the ability to assemble intelligence out of taps on providers' internal > connections > would require reverse engineering the ever changing protocols of all of those > providers.
This is somewhat less difficult than some people think. Various equipment manufacturers have implemented passive monitoring support on their interfaces specifically for these applications. You configure the interface to go into UP/UP state and to listen in a half duplex manner. This way you get the compatibility advantage of using standard network equipment to implement the interception, and so it will likely speak the same link-layer protocols the device being intercepted speaks. (E.g. here is some of the relevant documentation for Juniper: http://kb.juniper.net/InfoCenter/index?page=content&id=KB23036 and https://www.juniper.net/techpubs/en_US/junos11.2/topics/concept/flowmonitoring-passive-overview-solutions.html ) A lot of the mechanisms— the protocols, techniques, equipment features— for mass surveillance are easily visible to the public but the things visible to the public are all technical minutia dealing with the practical engineering challenges (Like the one you raise here— how the heck do you keep up with the ever changing layer 1/2 protocols used by service providers) that most people wouldn't even think to ask about. Using commodity hardware gets you compatibility, lower costs, and fast deployment. Even though budgets for massive surveillance no doubt allow for all kinds of specialized hardware— you can get more of it faster if you use commodity stuff with a few tweaks where you can. Here's another tidbit in public docs: Another challenge in implementing massive surveillance is the sheer volumes of traffic involved. People do seem to be aware of this one, but they argue that it makes the task impossible.... but there are few technical challenges which can't be solved by the suitable application of ingenuity and money. (_Lots_ of money: but keep in mind "defense" spending is just on another order of magnitude from regular spending. How much does a fighter jet cost? A one time use smart munition? How much more valuable is a good network tap than these devices? In light of that— how much is a fair defense industry price for one?) One way that the traffic volume problems gets solved is to realize that the vast majority of traffic is uninteresting. If you can rapidly filter the traffic you can throw out the 99% of uninteresting stuff and capture all of the rest. Filtering is, of course, computationally expensive— but it turns out that the power of 'commodity' technology can come to the rescue again: The same standard networking equipment that you need to speak the L1/L2 protocols on your optical taps also has built in line-rate packet filtering with scalability to millions of filter criteria (at no extra cost! :) ). The filtering in these devices has not historically been DPI grade: you can do stateless range/prefix matches on the packet headers, not free-form regex (although this is changing and the latest generation of hardware is more powerful— the need for NAT everywhere, if nothing else, is mandating it). But, if you can update those filters very fast— say, in under 50ms— then it doesn't matter that the filters aren't very powerful: Configure the filters to catch all known interesting hosts, the beginning of every new connection, and some small fraction (say, 1:10000 of all packets) and then feed that data to analysis systems which trigger updates to the filters when they spot something interesting. They only need to be powerful enough to limit a terabit of traffic to tens of gigabits, and that level of filtering can be accomplished just on 5-tuples.. You can go even further, then, by having two sets of filters with a delay line— say implemented using the >100ms of delay-product packet buffers in high end commodity networking hardware— in between them. The first set of filters catches enough so that your analysis systems can identify and track interesting flows, and by the time the traffic makes it through the delay line the second set of filters has been updated to capture the entirety of the interesting flow. ... though the persistence of traffic and the delay created by the TCP three way handshake make going this far not terribly necessary. Of course, using filtering in this way would require a protocol between the network elements and the analysis systems so that they could rapidly and dynamically 'task' the filters like you task surveillance satellites... And it sure would be convenient if the protocol was standardized so you could get many kinds of devices speaking it. ... something like: http://tools.ietf.org/html/draft-cavuto-dtcp-03 (and here is one vendor's helpful documentation on applying it, https://www.juniper.net/techpubs/en_US/junos/information-products/topic-collections/nce/lawful-intercept-flow-tap/lawful-intercept-using-flow-tap.pdf ) I've been continually amazed at how poorly the public has been doing at figuring out the mechanisms used for this stuff— You don't need some insider to tell you how it works, you could have just looked up the authors of packet interception standards and looked for people people who worked for major providers and advertise TS/SCI clearance on their resumes— then put together the pieces. People have observed that building high-tech infrastructure makes keeping secrets very costly but then fail to notice all of the completely open building blocks than just require the suitable application of big money. You're obviously not going to find smoking guns— things like production system core-dumps showing that which newspapers were being targeted for total surveillance— because the people building this stuff are not idiots, and it's not customary to publish that information generally. ... but the underlying technology and documentation needed to build this infrastructure, or at least the parts built out of commodity hardware. You absolutely can find that if you bother looking. But at the end of the day the technical details behind how it works really don't matter much. Just assume that a hostile party is at least passively monitoring all network links that aren't completely in your control and you'll be making the right kind of security decisions regardless of the technical minutia. -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
