Re: [tor-dev] How to distribute Tor with other software
Greetings, olde thread resurrection: Earlier Meejah pointed out that the tor control port can be used to create tor hidden services. Now that tor trac ticket #11291 (https://trac.torproject.org/projects/tor/ticket/11291) is resolved this will actually be usable? There are deployment issues. The program must run as a user who is a member of the tor group in order to read the hostname file and thus retrieve the onion address. With the current upstream master tor you can use the HiddenServiceDirGroupReadable option, like this: SETCONF hiddenservicedir=/var/lib/tor-alpha-hidden-services/hiddenService01 hiddenserviceport="80 127.0.0.1:8080" HiddenServiceDirGroupReadable=1 This should result in the hostname file being readable by the tor group. Is anyone else working on tor hidden service file sharing and or tor controller written in golang? I've implemented a golang api for creating and destroying tor hidden services using the control port... and a golang commandline tool for sharing directories via http + tor hidden service similar to carml pastebin or onionshare. If we write these tools in golang then we can use a deterministic build system much like the recent gitian golang deterministic builds for golang pluggable transports. Cheers, David signature.asc Description: Digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Using Shadow to increase agility of Sponsor R deliverables
On 20 Nov (14:45:12), Rob Jansen wrote: > Hi David, Roger, Hello! > > I think it would be great if we could show off some HS performance > improvements at the next Sponsor R PI meeting in January, both as a way to > show that we are making progress on the performance front and also to help > demonstrate our agility and how quickly we can get results on new designs. > I’m here to advocate the use of Shadow to help in this regard. > > So far, the list of deliverable that we could possibly show improvements is > quite small: > https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorR#Tickets > > From the list of completed tickets, the only one that stands out to me as > providing a performance boost is #13211 (Allow optimistic data on connections > to hidden services): > https://trac.torproject.org/projects/tor/ticket/13211 > > This seems like a somewhat small change; despite that, do we think this may > be something worth simulating to verify it works as expected and understand > the extent to which it reduces time to first byte for HS downloads? > > Are there other HS performance improvements that we think may be ready by > January? Here is what we (Karsten, George and me) are up to on that front for January. (Please guys, feel free to fill the blank if I am missing anything). We are aware of the january deadline so we've split up the work in two parts. George/Karsten are working on a proposal to gather HS statistics on the real Tor network to answer some questions that are presented here. https://people.torproject.org/~karsten/volatile/238-hs-relay-stats-2014-11-20.txt On my part, I have a chutney network with an HS and clients that fetch data on it. I'm currently working on instrumenting the HS subsystem so we can gather performance data and analyze it for meaningful pointers on where are the contention points, confirm expected behaviors, etc... I'll begin soon updating the following ticket with more information on the work I'm doing. (I'm in Boston right now collaborating with Nick for the week so things are a bit more slow on this front until monday). https://trac.torproject.org/projects/tor/ticket/13792 This could be used also with shadow I presume. Since the deadline is near us, I choose chutney for simplicity reasons here. I'll have a talk with Nick tomorrow on how we can possibly have this instrumentation upstream (either logs, controller event or/and tracing). Please note that Karsten is helping everyone here for both parts! :). On the host side of HS (meaning the HS relay itself), we have profiled an HS service that is being hammered with hundreds of connections. The client has been fixed to handle that also btw (#13698). I've talked this one out with Nick also and we have an idea for a solution that is in the mention in the ticket. Fixing that would in theory improve quite a bit the host side performance of HS. You can find the info here. https://trac.torproject.org/projects/tor/ticket/13739 Things are going forward, we still have some work ahead to gather the HS performance baseline and start trying to improve it. I'm fairly confident that the performance statistics in a private network will give us a good insight on the current situation. Feel free to propose anything that could be useful to make this thing more efficient/faster/useful :). Cheers! David > > Best, > Rob signature.asc Description: Digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] Using Shadow to increase agility of Sponsor R deliverables
Hi David, Roger, I think it would be great if we could show off some HS performance improvements at the next Sponsor R PI meeting in January, both as a way to show that we are making progress on the performance front and also to help demonstrate our agility and how quickly we can get results on new designs. I’m here to advocate the use of Shadow to help in this regard. So far, the list of deliverable that we could possibly show improvements is quite small: https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorR#Tickets From the list of completed tickets, the only one that stands out to me as providing a performance boost is #13211 (Allow optimistic data on connections to hidden services): https://trac.torproject.org/projects/tor/ticket/13211 This seems like a somewhat small change; despite that, do we think this may be something worth simulating to verify it works as expected and understand the extent to which it reduces time to first byte for HS downloads? Are there other HS performance improvements that we think may be ready by January? Best, Rob ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] high latency hidden services
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 09/11/14 18:33, Mansour Moufid wrote: > Has there been research on integrating high-latency message > delivery protocols with the hidden service model of location > hiding? The SecureDrop or Pynchon Gate protocols sound like good > starting points. I would love to participate, and encourage > everyone to start in this direction (in your copious free time ;). This issue has just come up on the guardian-dev list, so we're moving the conversation over here. Context quoted below. On Thu, Nov 20, 2014, at 09:46 AM, Michael Rogers wrote: > On 20/11/14 14:21, Nathan of Guardian wrote: >>> If we simply use Tor as a low-latency transport for >>> asynchronous messaging then we're limited to Tor's threat >>> model, i.e. we can't prevent traffic confirmation attacks. If >>> we revive one of the remailers or build a new system then we're >>> limited to a small number of users, i.e. a small anonymity set. >>> So ideally we'd find some way of adding high-latency mix-like >>> features to Tor. >> >> How much difference in latency are we talking about? Can we just >> introduce some sort of randomness or delay into our existing >> stacks/protocols? > > If we add delays at the application layer then those delays will > be the same all along the Tor circuit. So from the point of view of > an adversary doing a traffic confirmation attack against Tor, the > delays are irrelevant: the adversary sees the same pattern of > delays at both ends of the circuit, so the ends are still > correlated with each other. > > To decorrelate the traffic entering Tor from the traffic leaving > Tor we need to delay the traffic at each hop. Ideally we'd go > further than that and decouple high-latency traffic from circuits, > so that traffic could enter Tor on one circuit and leave on another > circuit, long after the first circuit was closed. But that's a much > harder problem than adding a delay at each hop, I think. > >>> Done right, this could provide a large anonymity set for the >>> high-latency users and improve the traffic analysis resistance >>> of Tor for the low-latency users at the same time, by providing >>> a pool of latency-insensitive traffic to smooth out the bursty >>> low-latency traffic between relays. >> >> I think this really makes the case, why a native Tor-based >> messaging channel/layer/link/substrate should be implemented. > > Great! Maybe we should move this discussion to the thread on > tor-dev that Mansour Moufid started recently? Cheers, Michael -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBCAAGBQJUbgMZAAoJEBEET9GfxSfMWegIAL/IQj7uOVSqd3kG+SSj/bCx RxyAkYmvS9josfDdtyOqDEJ3wMUcjMK313SPOA6vFWkkDYPqkbrXPwWCFMLJK/2m qVJ/ZItrno0RiOpTBEij2s7/VYW5ECzYjZUe3+O8lveYLz6k7IXXRiVkJt3y1oFn ze2Hl4XqTi7/rwaE9Q6JPfMBk4zcH4POtUaEYdpDME60QQQ8HRn0s2W9QlWihvrT ALhBqWXqZAZwEw1iKokT4XG/6HhGSs0WKgL53+wjfHTb6UdRTCas+nc/YuBuOVun 45OWmI8DuUG6YgxT5NEHKUL+OGts4ikoJ2TZwZ7Xjhd4zrgU+bZXTXDOQgUmJTY= =rcBt -END PGP SIGNATURE- ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Feedback on obfuscating hidden-service statistics
On 20/11/14 13:42, George Kadianakis wrote: > "A. Johnson" writes: > > George and I have been working on a small proposal to add two > hidden-service related statistics: number of hidden services and > total hidden-service traffic. Great, I’m starting to focus more on this project now. Well, actually I’m going on a trip for a week today, but *then* I’m focusing more on this project :-) >>> >>> Sounds great! We're meeting every Tuesday at 16:00 UTC in #tor-dev. >>> Feel free to drop by. >> >> Excellent. I won’t be there this coming Tuesday, but I’ll be there the next >> Tuesday. >> >>> Replicas mean that each descriptor is stored under two identifiers, so >>> that's two places. Further, descriptor identifiers change once per >>> day, so during a 24-hour period, there are up to four descriptor >>> identifiers for a hidden service. >> >> That makes sense. It would be nice if the statistics would allow you >> to identify how long (i.e. how many hour periods) each descriptor was >> observed being published. That would allow us to figure out if there >> are lots of short-lived services or fewer long-lived >> services. Publishing statistics every hour would pretty much take care >> of this. If you are really set on 24 hours, then perhaps you could add >> the total number of published descriptors in addition to the number of >> *unique* published descriptors. >> >> Also, my suggestion about using additive noise applies equally well to >> the descriptor statistics. And multiplicative noise is a *bad idea* if >> you don’t have some adjustment for small values (e.g. 10% noise of a 0 >> value is 0, and 10% of 1 is only 0.1). >> >>> We have been thinking about many more hidden-service related >>> statistics in a separate document. We're currently discussing whether >>> we should turn it into a tech report, because we'll probably not want >>> to implement most of those statistics. If you have remarks or more >>> ideas, please feel free to edit the document. We're going to have a >>> public review round for this, too, but that might not happen in the >>> next week or two. >>> >>> https://etherpad.wikimedia.org/p/hs_stats_78281091 >> >> Great! I think we should go for at least a little more data in the >> current proposal (what is the timeline for this, btw?). I think we >> should come up with a list of statistics we might imagine gathering >> and identify the subset of those that we’re comfortable gathering at >> this point. For example, I think failure statistics is much more >> innocuous than other data, and those would be very useful. For >> example, they would help us understand how to improve the protocol is >> failing, and it might help us identify misuse of hidden services >> (e.g. by botnets clients stupidly looking for non-existent descriptors >> or by malicious crawlers attempting to brute force descriptors). So >> here are some ideas: >> 1. Number of fetch requests for descriptors that don’t exist (number of >> fetch requests that do succeed would of course be very useful as well) >> 2. Number of descriptor publishes to the wrong HSDir (actually I suspect >> that the HSDir doesn’t check this and wants to be accepting of any publish) >> 3. Number of rendezvous circuits that never connect (from the RP >> perspective) >> 4. Number of rendezvous circuits on which no data cells are ever sent >> > > (CC'ed [tor-dev]) Thanks, George, for moving the discussion here. Here's the latest proposal draft where I incorporated Aaron's suggestions: https://gitweb.torproject.org/user/karsten/torspec.git/blob/refs/heads/hs_stats:/proposals/238-hs-relay-stats.txt If people on this list have more feedback, please reply here. Thanks! All the best, Karsten > Thanks for the input Aaron! > > The timeline here is that we are hoping the proposal _and_ the > implementation to be ready by mid-December. Then we are hoping that we > can deploy the code to a few relays so that we have some data by January. > > So, time is tight. > > I'm currently OK with the two statistics in: > https://people.torproject.org/~karsten/volatile/238-hs-relay-stats.txt > > I feel that any other statistics will need to be carefully analyzed. > We should add the ideas you mentioned in the etherpad, and get them > included in the tech report (which we are also hoping to have ready in > some form by mid-January). > > The tech report is supposed to contain and analyze most of the HS > statistics we can think of. It will likely contain many stats that we > will never do, but also some stats that might be a good idea. The good > ones we should eventually integrate to the Tor proposal and write code > for. > >>> Thanks for the very valuable input! Let me know if the following >>> draft looks okay, and I'll start another thread on tor-dev@. >>> >>> https://people.torproject.org/~karsten/volatile/238-hs-relay-stats-2014-11-20.txt >> >> "Lab(\epsilon/C)” -> "Lap(\epsilon/C)” (that was my mistake. I think
Re: [tor-dev] Feedback on obfuscating hidden-service statistics
"A. Johnson" writes: George and I have been working on a small proposal to add two hidden-service related statistics: number of hidden services and total hidden-service traffic. >>> >>> Great, I’m starting to focus more on this project now. Well, >>> actually I’m going on a trip for a week today, but *then* I’m >>> focusing more on this project :-) >> >> Sounds great! We're meeting every Tuesday at 16:00 UTC in #tor-dev. >> Feel free to drop by. > > Excellent. I won’t be there this coming Tuesday, but I’ll be there the next > Tuesday. > >> Replicas mean that each descriptor is stored under two identifiers, so >> that's two places. Further, descriptor identifiers change once per >> day, so during a 24-hour period, there are up to four descriptor >> identifiers for a hidden service. > > That makes sense. It would be nice if the statistics would allow you > to identify how long (i.e. how many hour periods) each descriptor was > observed being published. That would allow us to figure out if there > are lots of short-lived services or fewer long-lived > services. Publishing statistics every hour would pretty much take care > of this. If you are really set on 24 hours, then perhaps you could add > the total number of published descriptors in addition to the number of > *unique* published descriptors. > > Also, my suggestion about using additive noise applies equally well to > the descriptor statistics. And multiplicative noise is a *bad idea* if > you don’t have some adjustment for small values (e.g. 10% noise of a 0 > value is 0, and 10% of 1 is only 0.1). > >> We have been thinking about many more hidden-service related >> statistics in a separate document. We're currently discussing whether >> we should turn it into a tech report, because we'll probably not want >> to implement most of those statistics. If you have remarks or more >> ideas, please feel free to edit the document. We're going to have a >> public review round for this, too, but that might not happen in the >> next week or two. >> >> https://etherpad.wikimedia.org/p/hs_stats_78281091 > > Great! I think we should go for at least a little more data in the > current proposal (what is the timeline for this, btw?). I think we > should come up with a list of statistics we might imagine gathering > and identify the subset of those that we’re comfortable gathering at > this point. For example, I think failure statistics is much more > innocuous than other data, and those would be very useful. For > example, they would help us understand how to improve the protocol is > failing, and it might help us identify misuse of hidden services > (e.g. by botnets clients stupidly looking for non-existent descriptors > or by malicious crawlers attempting to brute force descriptors). So > here are some ideas: > 1. Number of fetch requests for descriptors that don’t exist (number of > fetch requests that do succeed would of course be very useful as well) > 2. Number of descriptor publishes to the wrong HSDir (actually I suspect > that the HSDir doesn’t check this and wants to be accepting of any publish) > 3. Number of rendezvous circuits that never connect (from the RP > perspective) > 4. Number of rendezvous circuits on which no data cells are ever sent > (CC'ed [tor-dev]) Thanks for the input Aaron! The timeline here is that we are hoping the proposal _and_ the implementation to be ready by mid-December. Then we are hoping that we can deploy the code to a few relays so that we have some data by January. So, time is tight. I'm currently OK with the two statistics in: https://people.torproject.org/~karsten/volatile/238-hs-relay-stats.txt I feel that any other statistics will need to be carefully analyzed. We should add the ideas you mentioned in the etherpad, and get them included in the tech report (which we are also hoping to have ready in some form by mid-January). The tech report is supposed to contain and analyze most of the HS statistics we can think of. It will likely contain many stats that we will never do, but also some stats that might be a good idea. The good ones we should eventually integrate to the Tor proposal and write code for. >> Thanks for the very valuable input! Let me know if the following >> draft looks okay, and I'll start another thread on tor-dev@. >> >> https://people.torproject.org/~karsten/volatile/238-hs-relay-stats-2014-11-20.txt > > "Lab(\epsilon/C)” -> "Lap(\epsilon/C)” (that was my mistake. I think > having the added noise both parameterized and included in the reported > statistics is an idea worth thinking about. Making it a parameter > allows you to easily change it without upgrading. Including it in the > statistics would allow us to correct better for noise if different > relays might be adding different amounts of noise due to inconsistent > opinions of the noise parameter (if this should never happen, then I > guess this wouldn’t be necessary). > > So again, sorry that I’m not g
