Re: Tor Project infrastructure updates in response to security breach
Hi Roger, Thanks for the detailed explanation. It's always interesting to hear about how other go into the verification route when a compromise happens. Do you know the nature of the compromise? Was it against Tor itself or one of the other services running on the Directory Authorities? Just curious, as it sounds like each of the DA was running a different set of apps, but perhaps I read more into that then was said. Also, is there a need for hardware to be apart to physically partition services (i.e. svn,git,dns)? Or do you guys already have that covered? Cheers, Harry Roger Dingledine wrote: On Wed, Jan 20, 2010 at 04:43:44PM -0500, Roger Dingledine wrote: In early January we discovered that two of the seven directory authorities were compromised (moria1 and gabelmoo), along with metrics.torproject.org Here are some more technical details about the potential impacts, for those who want to know more about Tor's innards: - #1: Directory authority keys Owning two out of seven directory authorities isn't enough to make a new networkstatus consensus (you need four for that), but it means you've only got two more to go. We've generated new v3 long-term identity keys for these two authorities. The old v3 long-term identity keys probably aren't compromised, since they weren't stored on the affected machines, but they signed v3 signing keys that are valid until 2010-04-12 in the case of moria1 and until 2010-05-04 in the case of gabelmoo. That's still a pretty big window, so it's best to upgrade clients away from trusting those keys. You should upgrade to 0.2.1.22 or 0.2.2.7-alpha, which uses the new v3 long-term identity keys (with a new set of signing keys). - #2: Relay identity keys We already have a way to cleanly migrate to a new v3 long-term identity key, because we needed one for the Debian weak RNG bug: http://archives.seul.org/or/announce/May-2008/msg0.html But we don't have a way to cleanly migrate relay identity keys. An attacker who knows moria1's relay identity key can craft a new descriptor for it with a new onion key (or even a new IP address), and then man-in-the-middle traffic coming to the relay. They wouldn't be able to spoof directory statements, or break the encryption for further relays in the path, but it still removes one layer of the defense-in-depth. Normally there's nothing special about the relay identity key (if you lose yours, just generate another one), but relay identity keys for directory authorities are hard-coded in the Tor bundle so the client can detect man-in-the-middle attacks on bootstrapping. So we abandoned the old relay identity keys too. That means abandoning the old IP:port the authorities were listening on, or older clients will produce warn messages whenever they connect to the new authority. Older Tor clients can now take longer to bootstrap if they try the abandoned addresses first. (You should upgrade.) - #3: Infrastructure services Moria also hosted our git repository and svn repository. I took the services offline as soon as we learned of the breach -- in theory a clever attacker could give out altered files to people who check out the source, or even tailor his answers based on who's doing the git update. We're in pretty good shape for git though: the git tree is a set of hashes all the way back to the root, so when you update your git tree, it will automatically notice any tampering. As explained in the last mail, it appears the attackers didn't realize what they broke into. We had already been slowly migrating Tor services off of moria (it runs too many services for too many different projects), so we took this opportunity to speed up that plan. A friendly anonymous sponsor has provided a pile of new servers, and git and svn are now up in their new locations. The only remaining Tor infrastructure services on moria are the directory authority, the mailing lists, and a DNS secondary. - #4: Bridge descriptors The metrics server had an archive of bridge descriptors from 2009. We used the descriptors to create summary graphs of bridge count and bridge usage by country, like the ones you can see at http://metrics.torproject.org/graphs.html So it's conceivable that some bad guy now has a set of historical bridge data -- meaning he knows addresses and public keys of the bridges, and presumably some of the bridges are still running at those addresses and/or with those public keys. He could use this information to help governments or other censors prevent Tor clients from reaching the Tor network. I'm not actually so worried about this one though, because a) we didn't have that many bridges to begin with in 2009 (you should run a bridge!), b) there seems to be considerable churn in our bridges, so last year's list doesn't map so well to this year's list), and c) we haven't been doing a great job lately at keeping China from learning bridges as it is. Hope that helps to explain, --Roger
Re: Tor Project infrastructure updates in response to security breach
Thus spake Paolo Palmieri (palma...@gmx.it): would it make sense to sign the torbutton xpi's? Actually, I've always been quite amazed by the fact that TorButton's .xpi (binary?) files are not signed. I'd really like to see this implemented in the future. Just as in the Tor repo, I gpg sign the Torbutton git tags. I also gpg sign .xpis, but have been sloppy about posting them publicly. As for actual Firefox-compatible builtin xpi signatures, the last time I looked into those they were exceedingly complicated and needed a special Code Signing Certificate, which required me bending over and paying Verisign or some other SSL Mafia Member a lot of money ($200-500/yr) to examine my rectum for a while. Maybe the Tor Project can get one of these for me, but I am not certain its really worth it. I suppose I could also create a rogue code signing certificate and provide that over SSL for people to install, but then I wonder if vanilla Firefox will reject my XPIs then because they are signed, but with an invalid cert. For now, I think the right answer is Fetch it over SSL or Check the git/gpg sig. -- Mike Perry Mad Computer Scientist fscked.org evil labs pgpG7EGSQVvmT.pgp Description: PGP signature
Re: Tor Project infrastructure updates in response to security breach
on Thu, Jan 21, 2010 at 02:09:42PM -0800, Mike Perry wrote: For now, I think the right answer is Fetch it over SSL Fetch it over SSL from addons.mozilla.org (the Mozilla Foundation obviously did bend over) /marcel signature.asc Description: Digital signature
Re: Tor Project infrastructure updates in response to security breach
Mike Perry wrote: I suppose I could also create a rogue code signing certificate and provide that over SSL for people to install, but then I wonder if vanilla Firefox will reject my XPIs then because they are signed, but with an invalid cert. I have a few of those laying around. I guess we could run some tests and find out? Best, Jake signature.asc Description: OpenPGP digital signature
Re: Tor Project infrastructure updates in response to security breach
Mike Perry wrote: Just as in the Tor repo, I gpg sign the Torbutton git tags. I also gpg sign .xpis, but have been sloppy about posting them publicly. snip For now, I think the right answer is Fetch it over SSL or Check the git/gpg sig. Could you make a point of publicly posting the .xpi gpg signatures along with the .xpis? I have never liked the method of downloading the extensions via the browser and installing all in one step. I prefer to download the extension, convince myself it is authentic (such as gpg), possibly install it locally in a test accound, and finally install it locally in the account(s) where I intend to use it. At present, the missing ingredient in being able to do that is not having a signature to verify against. So I'd much appreciate being able to get the signature w/o having to figure out git. Particularly if that signature has already been created. Thanks, Jim *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Tor Project infrastructure updates in response to security breach
You should upgrade to Tor 0.2.1.22 or 0.2.2.7-alpha: https://www.torproject.org/download.html.en In early January we discovered that two of the seven directory authorities were compromised (moria1 and gabelmoo), along with metrics.torproject.org, a new server we'd recently set up to serve metrics data and graphs. The three servers have since been reinstalled with service migrated to other servers. We made fresh identity keys for the two directory authorities, which is why you need to upgrade. Moria also hosted our git repository and svn repository. We took the services offline as soon as we learned of the breach. It appears the attackers didn't realize what they broke into -- just that they had found some servers with lots of bandwidth. The attackers set up some ssh keys and proceeded to use the three servers for launching other attacks. We've done some preliminary comparisons, and it looks like git and svn were not touched in any way. We've been very lucky the past few years regarding security. It still seems this breach is unrelated to Tor itself. To be clear, it doesn't seem that anyone specifically attacked our servers to get at Tor. It seems we were attacked for the cpu capacity and bandwidth of the servers, and the servers just happened to also carry out functions for Tor. We've tried to address the most common questions below. * Does this mean someone could have matched users up to their destinations? No. By design, Tor requires a majority of directory authorities (four in this case) to generate a consensus; and like other relays in the Tor network, directory authorities don't know enough to match a user and traffic or destination. * Does this mean somebody could have changed the Tor source? No, we've checked the source. It does mean you should upgrade so your client knows about all the currently valid directory authorities. * Does this mean someone could have learned more about Tor than an ordinary user? Since our software and specifications are open, everyone already has access to almost everything on these machines... except some old bridge descriptors, which we give out only in small batches as entry points for blocked clients. * Can I trust Tor's security? We've taken steps to fix the weaknesses identified and to harden our systems further. Tor has a track record of openness and transparency, with its source code and specifications and also with its operations. Moreover, we're disclosing breaches such as this so you can monitor our status. You shouldn't assume those who don't disclose security breaches never have any! --Roger signature.asc Description: Digital signature
Re: Tor Project infrastructure updates in response to security breach
On Wed, Jan 20, 2010 at 04:43:44PM -0500, Roger Dingledine wrote: In early January we discovered that two of the seven directory authorities were compromised (moria1 and gabelmoo), along with metrics.torproject.org Here are some more technical details about the potential impacts, for those who want to know more about Tor's innards: - #1: Directory authority keys Owning two out of seven directory authorities isn't enough to make a new networkstatus consensus (you need four for that), but it means you've only got two more to go. We've generated new v3 long-term identity keys for these two authorities. The old v3 long-term identity keys probably aren't compromised, since they weren't stored on the affected machines, but they signed v3 signing keys that are valid until 2010-04-12 in the case of moria1 and until 2010-05-04 in the case of gabelmoo. That's still a pretty big window, so it's best to upgrade clients away from trusting those keys. You should upgrade to 0.2.1.22 or 0.2.2.7-alpha, which uses the new v3 long-term identity keys (with a new set of signing keys). - #2: Relay identity keys We already have a way to cleanly migrate to a new v3 long-term identity key, because we needed one for the Debian weak RNG bug: http://archives.seul.org/or/announce/May-2008/msg0.html But we don't have a way to cleanly migrate relay identity keys. An attacker who knows moria1's relay identity key can craft a new descriptor for it with a new onion key (or even a new IP address), and then man-in-the-middle traffic coming to the relay. They wouldn't be able to spoof directory statements, or break the encryption for further relays in the path, but it still removes one layer of the defense-in-depth. Normally there's nothing special about the relay identity key (if you lose yours, just generate another one), but relay identity keys for directory authorities are hard-coded in the Tor bundle so the client can detect man-in-the-middle attacks on bootstrapping. So we abandoned the old relay identity keys too. That means abandoning the old IP:port the authorities were listening on, or older clients will produce warn messages whenever they connect to the new authority. Older Tor clients can now take longer to bootstrap if they try the abandoned addresses first. (You should upgrade.) - #3: Infrastructure services Moria also hosted our git repository and svn repository. I took the services offline as soon as we learned of the breach -- in theory a clever attacker could give out altered files to people who check out the source, or even tailor his answers based on who's doing the git update. We're in pretty good shape for git though: the git tree is a set of hashes all the way back to the root, so when you update your git tree, it will automatically notice any tampering. As explained in the last mail, it appears the attackers didn't realize what they broke into. We had already been slowly migrating Tor services off of moria (it runs too many services for too many different projects), so we took this opportunity to speed up that plan. A friendly anonymous sponsor has provided a pile of new servers, and git and svn are now up in their new locations. The only remaining Tor infrastructure services on moria are the directory authority, the mailing lists, and a DNS secondary. - #4: Bridge descriptors The metrics server had an archive of bridge descriptors from 2009. We used the descriptors to create summary graphs of bridge count and bridge usage by country, like the ones you can see at http://metrics.torproject.org/graphs.html So it's conceivable that some bad guy now has a set of historical bridge data -- meaning he knows addresses and public keys of the bridges, and presumably some of the bridges are still running at those addresses and/or with those public keys. He could use this information to help governments or other censors prevent Tor clients from reaching the Tor network. I'm not actually so worried about this one though, because a) we didn't have that many bridges to begin with in 2009 (you should run a bridge!), b) there seems to be considerable churn in our bridges, so last year's list doesn't map so well to this year's list), and c) we haven't been doing a great job lately at keeping China from learning bridges as it is. Hope that helps to explain, --Roger *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Tor Project infrastructure updates in response to security breach
In early January we discovered that two of the seven directory authorities were compromised (moria1 and gabelmoo), along with metrics.torproject.org, a new server we'd recently set up to serve metrics data and graphs. The three servers have since been reinstalled with service migrated to other servers. While the issue was resolved, could this of had an impact had they known what they broke into between the time of breach and time of discovery? Do we know how they broke in? *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Tor Project infrastructure updates in response to security breach
On Wed, Jan 20, 2010 at 11:12:29PM -0500, Peter Thoenen wrote: In early January we discovered that two of the seven directory authorities were compromised (moria1 and gabelmoo), along with metrics.torproject.org, a new server we'd recently set up to serve metrics data and graphs. The three servers have since been reinstalled with service migrated to other servers. While the issue was resolved, could this of had an impact had they known what they broke into between the time of breach and time of discovery? Yes, depending on how paranoid you want to get. I don't think they could have done anything particularly devious with the directory authority. We've got that pretty well sorted out with the distributed trust thing -- nothing moria1 does can rig the consensus by itself. So it's really a question of the services running. Moria was running a nameserver for torproject.org (still is), so they could send web requests elsewhere. If people check SSL certs, no problem (modulo the usual points about SSL not being perfect); if they don't check SSL certs, we hope they check package signatures. This risk isn't specific to our machines though -- your local ISP can lie to you about your DNS resolves, or some jerk could redirect our bgp record like how Pakistan stole Youtube for a few hours last year. It was also the mail host for @torproject.org, though most of the mails went off to other mail servers after that. So they could have read my mail. Most of my mail is public (and/or boring) anyway though. I could imagine that they might try to sneak in a commit to the git repository. We have a hook that mails all commits to the mailing list, and we watch that pretty well. But they could disable the hook during their commit. As I mentioned in the earlier mail, the git tree is made up of hashes, so they can't just modify it outright. I've looked over the 'git log' output, and didn't find anything odd. It might be neat to do an automated comparison of mails that made it to the mailing list vs commits to the git repository, if we wanted another layer of checking. Svn is less secure. It's just a database, and people can muck with it how they like. We've compared several of the svn repositories to backups, and nothing looked out of the ordinary. Good thing we moved Tor, Torbutton, BridgeDB, etc to git last year. The website wml files are still in svn and not git though, to make it easier for our volunteer translators; give us a holler if you find Tor sucks scribbled in some corner. :) If you want to scale up on the paranoid meter, you could imagine ssh client buffer overflows for the developers when we connected to it. That rabbit-hole goes as far as you like. Speaking of rabbit-holes, my gpg key is nearly a decade old and only 1024 bits. Sometime in the next little while I'm going to switch to a bigger one. Do we know how they broke in? As I understand it, we have a 450G disk image from one of the machines sitting somewhere in Canada, but not anywhere near any of the Tor people. The attacker(s) were sloppy, so we know some things like the name of the local-to-root exploit they used (which by its name works on a surprisingly wide spread of kernel versions... security is hard). I still don't know how they got in to moria originally, though. Too much was going on on that machine. --Roger *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Tor Project infrastructure updates in response to security breach
On Thu, Jan 21, 2010 at 12:25:08AM -0500, grarpamp wrote: It would be easier to just sign the git revision hashes at various intervals. Such as explicitly including the revision hash that each release is made from in the release docs itself. And then signing that release. That way everyone... git repo maintainers, devels, mirrors, users... can all verify the git repo via that signature. Of course the sig key material needs to be handled in a sanitary way, but still, it's the idea that matters. And git, not svn, would need to be the canonical repo committers commit to, etc. Thanks for Tor. We do sign the git repository for each release (stable and development). Do a git clone of Tor, and then 'git tag -l'. Saying the git hash of the release in the release notes is not a crazy notion though. --Roger *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Tor Project infrastructure updates in response to security breach
On Jan 21, 2010, at 6:25 AM, grarpamp wrote: As I wrote someone earlier... It would be easier to just sign the git revision hashes at various intervals. Such as explicitly including the revision hash that each release is made from in the release docs itself. And then signing that release. That way everyone... git repo maintainers, devels, mirrors, users... can all verify the git repo via that signature. Of course the sig key material needs to be handled in a sanitary way, but still, it's the idea that matters. And git, not svn, would need to be the canonical repo committers commit to, etc. This already happens. Clone the Tor repository, and you'll find a signed tag named tor-0.2.2.7-alpha. Use git tag -v tor-0.2.2.7-alpha to check for yourself. Sebastian *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Tor Project infrastructure updates in response to security breach
ok, cool. thx guys. would it make sense to sign the torbutton xpi's? and torsocks? perhaps since it all comes from the same git repo it isn't necessary. *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
Re: Tor Project infrastructure updates in response to security breach
would it make sense to sign the torbutton xpi's? Actually, I've always been quite amazed by the fact that TorButton's .xpi (binary?) files are not signed. I'd really like to see this implemented in the future. Thanks, Paolo *** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talkin the body. http://archives.seul.org/or/talk/
