Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Hello all, Just a quick heads up for where this is going: * 'GET_ONIONS' was changed to be 'GETINFO' subcommands for consistency. * Code is in 'needs_review', for the 2nd round. As a proof of concept I wrote: https://github.com/Yawning/onionwrap It's a dumb quick and dirty hack that works like this: $ TOR_CONTROL_PORT=9251 ./onionwrap -port=80,8080 godoc -http=%TADDR INFO: Created onion: 5rn73le64hri2zmp.onion:80 - 127.0.0.1:8080 [onionwrap blocks till killed] The %TADDR notation is a convenience thing for automation, there's also %TPORT, and %VPORT that get rewritten as appropriate. The onionwrap code is still incomplete, so I wouldn't recommend using it quite yet, especially since it: * Only supports creating one shot Onion Services. * Should support talking to the real server over AF_UNIX. * Needs documentation. * Requires a really experimental branch based on tor's master. * Maybe should use stem/txtorcon instead of bulb[0]. But as a proof of concept and a demonstration of the feature, I think it gets the point across. Thanks to special for inspiring me to write this. Regards, -- Yawning Angel [0]: Which is another quick and dirty hack I wrote, so I could write things like this. pgpEG1HqWaqA8.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Wed, 11 Mar 2015 02:35:10 + Yawning Angel yawn...@schwanenlied.me wrote: The code: https://github.com/Yawning/tor/compare/feature6411 The spec: https://github.com/Yawning/torspec/compare/feature6411 Minor updates to both over the course of yesterday, thanks to all that gave useful feedback and corrections. The notable changes are: * DiscardPK is now Flags=DiscardPK, to allow for. [0]: Onions added this way will remain tied to the control port connection that created them for now. That particular dead pony has been loaded onto the cart and shipped off to the glue factory, and is no longer available for beatings. * I lied. People that want to shoot themselves in the foot can now specify the Detach flag which explicitly unlinks the newly created HS from any control port instance. Detached Onion Services will persist till tor termination, or explicitly removed by DEL_ONION. * DEL_ONION now will allow the removal of Onion Services belonging to the current connection in addition to any Onion Service created with the Detach flag. * Added GET_ONIONS that returns a list of Onion Services belonging to the current connection in addition to all Onion Services created with the Detach flag. I strongly recommend ignoring the fact that the Detach flag and GET_ONIONS command exists, beyond reviewing my code and making sure they're implemented correctly. That said, I know multiple developers will probably write applications that implements cleanup as DEL_ONION ALL THE THINGS, stomping over unrelated Detach-ed services[0]. Unless something comes up, I plan to rebase/squash my feature branch for review sometime by early next week, so if people have strong opinions on this feature, they should speak up now. Regards, -- Yawning Angel [0]: I have my not a bug/wontfix ready for such situations. pgpZA9563xnv3.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Wed, 25 Feb 2015 13:51:59 +, carlo von lynX wrote: ... What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket. One way to do that would be to tie the hidden service to the existence of the PID of your app - just exec the app in the script after setting up the HS. (I seem to remember that being an option in some form already.) Alternatively tor could check whether the listener the HS is accessing is still open, and discard the HS when that is no longer the case. (Possibly new idea.) (And hopefully your application isn't giving extended authority to clients connecting from 127.0.0.1.) Andreas -- Totally trivial. Famous last words. From: Linus Torvalds torvalds@*.org Date: Fri, 22 Jan 2010 07:29:21 -0800 ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Wed, Feb 25, 2015 at 07:41:01AM +0100, Andreas Krey wrote: The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy. Yes, but as it stands AF_TOR is not the #1 API deployed in network applications. The majority of hidden services are $whatever configured to listen on port localhost:something and zero awareness of any tor router doing the rest of the work. Having to change hundreds of existing apps so that they can work with tor without having to edit torrc is a worse tradeoff than having to edit torrc. What is useful here is if I can use existing $app with existing tor router and just have a shell script drop the glue instructions into the tor unix socket. This allows to configure hidden services without editing neither torrc nor the code of the $app. A bit like Vidalia does it, but without messing with the torrc and without being Vidalia. Also, would you entrust your hidden service keys to a system-wide tor process? :-) It's like asking if I trust my bank to keep my money. If you don't trust your Tor process, try not using a computer. If you don't trust your cheap server hosting provider, don't put hidden services on cheap rented servers or virtual machines. But if you don't trust your OS, there is no place where you can hide your hidden service keys. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Hello, Not sure about the renaming thing. I'm open to doing so, but a lot of the user facing stuff (the torrc configuration variables etc) use the HS terminology, so I'm not sure if being inconsistent is good. On Wed, 25 Feb 2015 13:51:59 +0100 carlo von lynX l...@time.to.get.psyced.org wrote: On Wed, Feb 25, 2015 at 07:41:01AM +0100, Andreas Krey wrote: The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy. Yes, but as it stands AF_TOR is not the #1 API deployed in network applications. The majority of hidden services are $whatever configured to listen on port localhost:something and zero awareness of any tor router doing the rest of the work. Having to change hundreds of existing apps so that they can work with tor without having to edit torrc is a worse tradeoff than having to edit torrc. So, write a python script that pulls in txtorcon or stem, does the appropriate controller goo, and subprocess.call()s $whatever. Anyway now that master is 0.2.7.x, I'm looking to revisit this in the form of: * Renaming the commands if necessary. * Minor code cleanups (that occurred to me after I let it sit for a while). * Adding an argument to `ADD_EPH_HS` to suppress the private key being returned over the control port, when the caller requests that tor generate the keypair (for cases where the HS is truely oneshot and the user does not wish to preserve it). * Documentation. Regards, -- Yawning Angel pgpAaj3f82G9d.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
carlo von lynX l...@time.to.get.psyced.org writes:
I like your suggestion, and while we're bikeshedding ;) what if we use
the recently-proposed naming system and call it ADD_ONION?
Concerning the ephemerality of it, I can imagine services
being configured en passant by a cat socket from a shell
script or so, [..]
You still need to authenticate. I do like the simplicity, but it will be
a little more complex than that. I guess it's a bit of extra work to
keep such a connection around. But really, it's just storing a PID and
killing it when you're done.
It's still, I think, worth distinguishing somehow between an onion
service added via SETCONF (which will get written to disc, and written
to the torrc potentially) and the API Yawning has added that will vanish
if the tor instance is re-started (and has no hidden service dir at all).
Another option could always be added in the future, like
lifetime={controller,tor} or something if the goes away with
process makes it harder than necessary to use. My instincts still say
that controller connection lifetime is a good API, but that's not a
super compelling argument ;)
--
meejah
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Wed, 25 Feb 2015 05:06:37 +, carlo von lynX wrote: the advantages of that aren't obvious to me. why would i need to make every networking app hold the hand of its router to let it know it's still needed? You answer your question yourself: tor is on its way to becoming an AF_TOR - a networking esssential. making a hidden service could one day be as simple as doing listen() on an AF_TOR socket... The AF_TOR listener would go away with closing the listener socket as well (and thus is bound to the lifetime of the process); so binding a hidden service to the control connection is the obvious analogy. Whether tor as such should exit with loss of control connections is (more) debatable - if my app starts the tor process, I'd expect it to terminate when it detects that I'm no longer there. If it's a system/shared daemon we'd need to talk about how it comes into existence in the first place, and how it is shared. Also, would you entrust your hidden service keys to a system-wide tor process? :-) Andreas -- Totally trivial. Famous last words. From: Linus Torvalds torvalds@*.org Date: Fri, 22 Jan 2010 07:29:21 -0800 ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Concerning the ephemerality of it, I can imagine services
being configured en passant by a cat socket from a shell
script or so, [..]
On Tue, Feb 24, 2015 at 09:05:38PM +0400, meejah wrote:
You still need to authenticate. I do like the simplicity, but it will be
a little more complex than that. I guess it's a bit of extra work to
keep such a connection around. But really, it's just storing a PID and
killing it when you're done.
posix has user/group access rights for file sockets and linux has acls
on top of that. using a localhost socket is less secure, i agree.
It's still, I think, worth distinguishing somehow between an onion
service added via SETCONF (which will get written to disc, and written
to the torrc potentially) and the API Yawning has added that will vanish
if the tor instance is re-started (and has no hidden service dir at all).
that is too inflexible.. i may want non-ephemeral services activated
by a cat to a socket. i don't see the point in an artificial requirement
to have config things frozen into a text file.
Another option could always be added in the future, like
lifetime={controller,tor} or something if the goes away with
process makes it harder than necessary to use. My instincts still say
that controller connection lifetime is a good API, but that's not a
super compelling argument ;)
the advantages of that aren't obvious to me. why would i need to
make every networking app hold the hand of its router to let it
know it's still needed? do we do this for regular networking, too?
tor is on its way to becoming an AF_TOR - a networking esssential.
making a hidden service could one day be as simple as doing
listen() on an AF_TOR socket...
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Tue, 17 Feb 2015 02:22:54 +0400 meejah mee...@meejah.ca wrote: From my perspective, the entire point of this feature is to allow applications to use the system Tor (or, at least some already-running tor) to put their hidden services on. The design the way it is with more isolation than usual also allows for people that use the tails/whonix/or my control port tinfoil hat filter to potentially whitelist the commands as well, since there isn't any information disclosure (unless I screwed up the implementation). Modifying the filters is left up to the implementors (and I probably won't change or-ctl-filter to allow them since I don't use HSes[0]), but I view it as a step forward for including/using applications that use HSes into such environments. (Or, looking at it another way, if you don't want to share a tor instance with other applications, you can do that easily today -- you start up a tor, and can use __OwningControllerProcess as you see fit). The *only* reason that txtorcon by default launches a new tor process for the onion: endpoints is because nothing else will work reliably. I'd very much like the default to be that it adds a new onion using Yawning's API. This will still fall back to launch a new tor process, but if you know what you're doing you can have a system (or personal) Tor instance running (with cookie authentication) that can have hidden services added and removed from it. Obviously, this will be a lot faster than launching a new Tor (and less load or DirAuths). As a bonus, Yawning's API is nice and simple (versus SETCONF, which makes it super easy to nuke all the *other* hidden services you configured). What I would expect of people using this feature is that the process that's adding the hidden service is the very same one that's providing the service... (If you're doing something with a separate nginx or whatever process running your hidden-service, than you should know enough to be able to add a couple lines to your torrc and make a normal, permanent hidden-service). Or write a trivial script that prompts for a password, decrypts a HS key, and loads it into a tor instance over the control port. I assume if/when my branch gets merged into mainline tor that the controler APIs will support it in a easy to use manner. So, in that context, if your application code isn't perfect and might sometimes crash, definitely the right thing is to nuke the hidden service. If the app didn't even save its private keys before that happened, well, too bad. I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away. It seems a lose consensus is reached here that this is ok, so I'm going to leave the design as is and write the control-spec.txt patch. -- Yawning Angel [0]: The first HS I ever set up was when I finished my first pass implementation, and got the code to a working state. pgpBl_7Y6_Y46.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Fri, Feb 13, 2015, at 04:45 PM, Yawning Angel wrote: Yes, this means that if you run kittensomgmewmew.onion and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites, ...or the backdoor in the firmware on your hard disk... :) --dave ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Let me chime in on saying that this looks to me like a great development. I even imagine that in a couple of years most end-to-end encrypted services on the Internet may be using this interface, so for the sake of accessibility for future devs, I would suggest something totally superficial: On Sat, Feb 14, 2015 at 12:45:24AM +, Yawning Angel wrote: ADD_EPH_HS - Adds a new ephemeral hidden service. The ephemeral vs current behavior seems to be an aspect that might go away soon anyway, so I would leave the cryptic EPH out. Also, the idea that hidden services are hidden is a bit outdated - many are intentionally not hidden at all and only interested in the aspect of providing authenticated end- to-end encryption and/or anonymization for the calling entity, so HS isn't exactly an acronym which speaks a clear language. Thus I would think it will be easier for future generations of devs and sysadmins to grok this, if the functions were simply named ADD_SERVICE, REMOVE_SERVICE. Concerning the ephemerality of it, I can imagine services being configured en passant by a cat socket from a shell script or so, while the actual application was merely told to receive connections on localhost:something and has no awareness of Tor at all, like most Tor HS are, I presume. So in the spirit of Leif's suggestion I would make the dependency on a ControlPort link staying up the non-default. KUTGW! -- E-mail is public! Talk to me in private using Tor. torify telnet loupsycedyglgamf.onion DON'T SEND ME irc://loupsycedyglgamf.onion:67/lynX PRIVATE EMAIL http://loupsycedyglgamf.onion/LynX/OR FACEBOOGLE ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Mon, 16 Feb 2015 10:17:51 -0500 David Goulet dgou...@ev0ke.net wrote: [snip] A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process. Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say ControlListenAddress is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS? That's correct, though unless tor or the controller library has code to stomp on long dormant connections (which a casual look says there isn't), this shouldn't happen, because TCP/IP in itself has no idle timeout (See RFC 1122 4.2.3.6 regarding keep alives, which would also not cause HS loss, since the OS will respond to the probe). There may be certain broken middleboxes (loadbalancers etc) that stomp on long idle TCP connections, but anyone that is running a control port connection through such a thing, and sending RSA keying material in the clear, probably has bigger things to worry about. This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS? Yup, I don't see you need to leave stem running as being all that bad, since this is mostly targeted at managed applications (chat, filesharing, global leaks, etc). If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things. Regards, -- Yawning Angel pgpw3rUD3yJJF.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Mon, Feb 16, 2015 at 03:47:07PM +, Yawning Angel wrote: On Mon, 16 Feb 2015 10:17:51 -0500 David Goulet dgou...@ev0ke.net wrote: [snip] A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process. Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say ControlListenAddress is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS? That's correct, though unless tor or the controller library has code to stomp on long dormant connections (which a casual look says there isn't), this shouldn't happen, because TCP/IP in itself has no idle timeout (See RFC 1122 4.2.3.6 regarding keep alives, which would also not cause HS loss, since the OS will respond to the probe). There may be certain broken middleboxes (loadbalancers etc) that stomp on long idle TCP connections, but anyone that is running a control port connection through such a thing, and sending RSA keying material in the clear, probably has bigger things to worry about. This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS? Yup, I don't see you need to leave stem running as being all that bad, since this is mostly targeted at managed applications (chat, filesharing, global leaks, etc). If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things. First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications. I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash. However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. I'd rather see two modes: one as you describe, and another in which the ephemeral HS stays running until a new control port connection requests that it be stopped. To avoid allowing enumeration of running services, the stop command could require that the requestor already knows some details of the HS - either a cookie generated at creation time, or perhaps just the private key that was provided when it was started. This of course wouldn't result in crashed applications' HSes being cleaned up automatically, but having a few stale HSes sitting around isn't the end of the world. One approach for cleaning them up could be that tor could remove them automatically after it sees connection refused a few times. ~leif ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Mon, 16 Feb 2015 16:11:55 + Leif Ryge l...@synthesize.us wrote: [snippity] However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. I'd rather see two modes: one as you describe, and another in which the ephemeral HS stays running until a new control port connection requests that it be stopped. To avoid allowing enumeration of running services, the stop command could require that the requestor already knows some details of the HS - either a cookie generated at creation time, or perhaps just the private key that was provided when it was started. dgoulet suggested Detach=true as an optional argument, which is what the add side interface would look like if I did this. This of course wouldn't result in crashed applications' HSes being cleaned up automatically, but having a few stale HSes sitting around isn't the end of the world. One approach for cleaning them up could be that tor could remove them automatically after it sees connection refused a few times. I'm not quite sure how I feel about this yet. The code for doing all of this isn't that difficult, but I'd want to hear from a few more people about what the right thing to do here would be. Most importantly since the `ADD_EPH_HS` interface uses key/value pairs for the port/target now, this would be easy to add on at a later date even if it doesn't get included in the first iteration. Something to discuss at the dev-meeting if consensus hasn't been reached by then. Regards, -- Yawning Angel pgpfWLpkffCJ7.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 (CCing the hidden-services list.) On 16/02/15 16:11, Leif Ryge wrote: If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things. First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications. Seconded! I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash. As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot. Are there any such apps, and is it a good idea to support such apps (has the rest of the control protocol been examined for cross-controller data leaks, what happens if apps tread on each other's configuration changes, etc)? However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. Dormant processes are essentially free, so does this matter? Cheers, Michael -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBCAAGBQJU4kaeAAoJEBEET9GfxSfMP/oH/Aoel3gyOAtn5NrgKh6WRcFf 5TwPOElP/vL+XI5XrPRYJYczilQ2st/ZLu6nuULLvGoqbtDZ0VU23uyffPhypx87 n5hXPNYbXt7/tvJ42ULq509D1nRI9Xp39YOwPMt0Yw7RXWo2eB7eYd2n0tMXrdan 4hhxIqe21MXXiL9QGuf/MaToFRQP9TB2vNP5eZzS07WR1EvSN5TvBO5nZM9TRE4t daJ0mNPhL4v6gb0j0iVCzZJFZ424swOyqdOu5K7iPRWkMbacX9uXINzjUn8NWzIO hT6GK3dHsyhGjiWLQ0Dlpw1yIZ6vv5YCKAbYoS7mSS0U1NNSaouaT7zzR7+GIMo= =uVVW -END PGP SIGNATURE- ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Mon, 16 Feb 2015 19:35:58 + Michael Rogers mich...@briarproject.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 (CCing the hidden-services list.) (Wonder if my reply will bounce.) On 16/02/15 16:11, Leif Ryge wrote: If someone has a suggestion for an alternative interface that can handle applications crashing (possibly before they persist the list of HSes they need to clean up), applications that are just poorly written (and not cleaning up all the ephemeral HSes), and (optionally, though lacking this would be a reduction in features) limiting cross application HS enumeration, I'd be more inclined to change things. First, thanks for doing this! I think it's a great feature which will make it much easier to create new hidden service applications. Seconded! I like the idea of tying HS lifetime to the control port connection for the reasons you state, namely that cleanup is automatic when applications crash. As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot. Yep. I suspect that well behaved applications won't need this for the most part, but by defining things this way, it avoids unpleasant surprises for apps that aren't written well, or those that do try to share a tor instance. Are there any such apps, and is it a good idea to support such apps (has the rest of the control protocol been examined for cross-controller data leaks, what happens if apps tread on each other's configuration changes, etc)? I haven't looked at other concerns in depth, and AFAIK it's a huge problem area. My concerns in this area are more does my branch make the current situation any worse, rather than solve all the control port problems, and make sure this is totally a safe/fine/recommended thing to do (It's none of those things). If most if not all apps will set _OwningControllerProcess, the current behavior doesn't hurt either since the tor instance will die anyway, and on the off chance that someone writes a not-so-great app (heaven forbid), the right thing happens. However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. Dormant processes are essentially free, so does this matter? I have this mindset too. -- Yawning Angel pgpUVCOT7N8h3.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
As an app developer this strikes me as the right approach. But having said that, I wouldn't actually need this feature because Briar already uses __OwningControllerProcess to shut down Tor if the control connection is closed. I imagine the same would apply to any app that manages its own Tor process - so this feature would only be useful for apps that share a Tor process with other apps and communicate directly with it via the control port, rather than indirectly via a controller such as Orbot. This feature is going to change how tor controllers operate... and whether the tor controller is a separate process or an API that the application uses seems to have been overlooked in this last statement of yours. Here's an example of what I mean; In the case of Twisted Python applications... olde apps written a decade ago are made to work with Tor hidden services without a single line of code changing... because the endpoint is dynamically loaded and the controlling of tor is abstracted by this Twisted endpoint. pip install txtorcon twistd -n web --port onion:80 --path ~/public_html In this case... can twistd be said to manage it's own tor process? It certainly does not contain any Tor specific code whatsoever. It is managed in the same process but the details are abstracted away by the txtorcon controller API. Furthermore it is not limited to apps that share a tor process with other applications... Perhaps the computer has multiple instances of tor running; the Tor controller could be setup to connect to any of these control ports. This is an observation about a deployment issue more so than an API issue. However, it seems like in the case of applications which are not HS-specific this will necessitate keeping another process running just to keep the HS alive. responding to Leif's comment above: This seems like a false statement in the context of using txtorcon with this feature. The above txtorcon twistd example could be made to work with the tor control port and it would not imply keeping another process running. Furthermore, the above example currently relies on txtorcon launching it's own Tor process which I think is convenient but not so good for security isolation... and implies ambient authority, that is, why should the python process have that much control over the tor process? The python app only needs the privilege to create and remove it's Tor hidden service (principal of least privilege ftw). Dormant processes are essentially free, so does this matter? Yes, it seems like it doesn't matter much.. The local connection from the tor process would fail to connect to the local service. I'm not sure if there are other implications that are more important than this. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
From my perspective, the entire point of this feature is to allow applications to use the system Tor (or, at least some already-running tor) to put their hidden services on. (Or, looking at it another way, if you don't want to share a tor instance with other applications, you can do that easily today -- you start up a tor, and can use __OwningControllerProcess as you see fit). The *only* reason that txtorcon by default launches a new tor process for the onion: endpoints is because nothing else will work reliably. I'd very much like the default to be that it adds a new onion using Yawning's API. This will still fall back to launch a new tor process, but if you know what you're doing you can have a system (or personal) Tor instance running (with cookie authentication) that can have hidden services added and removed from it. Obviously, this will be a lot faster than launching a new Tor (and less load or DirAuths). As a bonus, Yawning's API is nice and simple (versus SETCONF, which makes it super easy to nuke all the *other* hidden services you configured). What I would expect of people using this feature is that the process that's adding the hidden service is the very same one that's providing the service... (If you're doing something with a separate nginx or whatever process running your hidden-service, than you should know enough to be able to add a couple lines to your torrc and make a normal, permanent hidden-service). So, in that context, if your application code isn't perfect and might sometimes crash, definitely the right thing is to nuke the hidden service. If the app didn't even save its private keys before that happened, well, too bad. I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away. -- meejah ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On 2/16/15 11:22 PM, meejah wrote: I guess to put another way: I can't see a use-case to keep the hidden- service around if the application that added it went away. +1 from globaleaks perspective -naif ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On 14 Feb (00:45:24), Yawning Angel wrote: Hey Yawning, great stuff btw! I have a questions below regarding meejah's comment and https://trac.torproject.org/projects/tor/ticket/6411#comment:32 Ephemeral hidden services are tied to the control port connection that created them. This means, that when the control connection goes away, so does the hidden service intro point. Closing client connections is left as an exercise for the application. [...] Design question that I want feedback on, primarily from control port library maintainers, and authors of applications that use the control port: * meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing. My motivations for strongly considering implementing this is that, I don't think that adding a command to enumerate hidden services is a good idea (SketchORChat has no business in seeing what hidden services JankORFileSharing is using), and that this will automate cleanup in the event of an application crashing, or not removing it's hidden services before terminating. Can you explain the rational for this? This email states in the unofficial-tor-spec-patch: A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process. Now, an HS bound to a control connection might be a good idea, I'm not 100% sure but I can see issues with this. Let's say ControlListenAddress is used, this means a TCP socket and it can timeout if no activity, so if that happens, I loose my HS? This also put quite a requirement on the user side to add an HS on its tor-ramdisk for instance but has to use a client that keeps the control connection opens for its lifetime?... How does that work with stem, it would have to keep the control connection open and the script using it can't quit else the socket gets closed by the OS? Thanks! David signature.asc Description: Digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Yawning Angel yawn...@schwanenlied.me writes: Cool! I added a quick try at using this in txtorcon, with an example; see the following branch: https://github.com/meejah/txtorcon/tree/yawning-feature-6411 (Most of that example code would actually end up in the endpoint.listen() code and be hidden from most users of the library, too. And would be the default instead of launch a new tor). This does not support `HiddenServiceAuthorizeClient` yet in any way (neither basic nor stealth). It may in the future, but coming up with a sane interface for it is way more pain than I am willing to self-inflict at the moment. What about a similar API: add + remove a client-key. And similarly, you could push a key yourself, or ask for a new one. The only difference would be including the onion-address. Like: ADD_EPH_HS_ADD_CLIENT_AUTH kittenmeowae41f nickname0 stealth NEW:BEST ADD_EPH_HS_ADD_CLIENT_AUTH kittenmeowae41f nickname1 basic NEW:BEST The return value would be the cookie (for basic) or both the cookie and the key-blob for stealth. Or NEW:BEST could be RSA1024:. like the other API? For deleting you'd pass the onion ID and the nickname I guess. Like: ADD_EPH_HS_DEL_CLIENT_AUTH kittenmeowae41f nickname1 * meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing. __OwningControllerProcess is similar, I guess, but instead ties to the whole lifetime of Tor (which sounds way easier to implement ;) Questions, comments, feedback appreciated, I think it's looking quite nice! -- meejah ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Hi Yawning, nice addition! As requested moving from irc to here. One gotcha to think about is that ADD_EPH_HS is using a variable number of positional arguments. This will limit your ability to expand this command in the future with new arguments. Also, I'd suggest renaming addrPort to addrTarget (to avoid making this sound restricted to a simple port). Other than that spec looks great. Looking forward to it! -Damian On Fri, Feb 13, 2015 at 4:45 PM, Yawning Angel yawn...@schwanenlied.me wrote: Hi, The Warning: DO NOT USE MY BRANCH YET, IT HAS HAD MINIMAL TESTING AND REVIEW. IT IS NOT DONE. IT WILL BROADCAST YOUR SECRETS TO THE NSA'S ORBITAL SPACE STATION. Trac Ticket: https://trac.torproject.org/projects/tor/ticket/6411 Branch: https://github.com/Yawning/tor/compare/feature6411 With that out of the way, I've been poking at implementing a way to add hidden services at runtime via the control port. I'm calling this ephemeral hidden services because the HS state is not written out to disk, and they disappear when the tor damon is shutdown. Yes, this means that if you run kittensomgmewmew.onion and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites, you could run your tor instance entirely in a ram disk, and load the HS key manually each time from a USB token you wear around your neck. What I've come up with looks like this: ADD_EPH_HS - Adds a new ephemeral hidden service. The syntax is: ADD_EPH_HS keytype:keyblob SP 1*(port SP addrPort) CRLF keytype = RSA1024, NEW keyblob = opaque ascii serialized key material, or an algorithm specifier, in the case of keytype == NEW. port = the VIRTPORT addrPort = the TARGET A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process. The currently supported keytypes and keyblobs are: RSA1024 - The keyblob contains the Base64 encoding of the PKCS#1 layout DER representation of a 1024 bit RSA private key. NEW - The tor instance should generate a key with the keytype specified in the keyblob parameter. A type of BEST will generate a key using the best supported algorithm. Returns: 250-ServiceID=serviceID - The hidden service's onion address without the .onion suffix. 250-PrivateKey=keytype:keyblob - The newly generated hidden service's private key, if a keytype of NEW was specified. 250 OK - Everything went well. Proper key management, is left as an exercise for the application invoking the controller command. DEL_EPH_HS - Removes an existing ephemeral hidden service. The syntax is: DEL_EPH_HS serviceID CRLF serviceID = the hidden service identifier, without the trailing .onion suffix The hidden service specified by the given service ID is removed, and the intro points are closed. Existing connections are left as is, and it is up to the application to close them if it wishes to do so. Examples: * Adding a new ephemeral hidden service, with a pre-generated RSA key: ADD_EPH_HS RSA1024:[Base64 blob] 80 127.0.0.1:8080 250-ServiceID=blahblahblahblah 250 OK Hidden service is reachable at blahblahblahblah.onion:80 * Adding a new ephemeral hidden service, tor generates a new RSA key: ADD_EPH_HS NEW:RSA1024 22 127.0.0.1:22 250-ServiceID=hogehogehogehoge 250-PrivateKey=RSA1024:[Base64 blob] 250 OK Hidden service is reachable at hogehogehogehoge.onion:22. At a later date, the value passed back as PrivateKey can be used to re-create the hidden service. * Add a new ephemeral hidden service, tor picks the algorithm: ADD_EPH_HS NEW:BEST 1245 192.168.1.5:23 250-ServiceID=hemohemohemohemo 250-PrivateKey=Ed25519:[Base64 blob] 250 OK Hidden service is reachable at hemohemohemohemo:23. (nb: Ed25519 key support is not actually implemented yet, and is used as an example of a better algorithm.) * Remove the hidden service configured in the first example: DEL_EPH_HS blahblahblahblah 250 OK Note: ADD_EPH_HS can take more than one VIRTPORT/TARGET pair at once, the examples only show one pair for brevity. This does not support `HiddenServiceAuthorizeClient` yet in any way (neither basic nor stealth). It may in the future, but coming up with a sane interface for it is way more pain than I am willing to self-inflict at the moment. Design question that I want feedback on, primarily from control port library maintainers, and authors of applications that use the control port: * meejah suggested that ephemeral hidden services should have their lifetime tied to the originating
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On 2/14/15 1:45 AM, Yawning Angel wrote: Hi, The Warning: DO NOT USE MY BRANCH YET, IT HAS HAD MINIMAL TESTING AND REVIEW. IT IS NOT DONE. IT WILL BROADCAST YOUR SECRETS TO THE NSA'S ORBITAL SPACE STATION. Trac Ticket: https://trac.torproject.org/projects/tor/ticket/6411 Branch: https://github.com/Yawning/tor/compare/feature6411 I'm fine with the proposal. That's an important part of a set of tickets designed to enable the use of a Tor integration without ever touching the filesystem within third party application and application controllers (such as TxTorCon/ORbot) . This complement the already closed #13865 (https://trac.torproject.org/projects/tor/ticket/6411) and the yet-to-be-discussed #14899 (Enable Tor to work without using filesystem for cached files https://trac.torproject.org/projects/tor/ticket/14899). When #6411 will be integrated and #14899 will be implemented: - Tor AppArmor profile can be imporved by completely disabling filesystem read/write (when integrated with a third part app) - Third party App can fully use Tor by keeping all it's configuration directive, keys (for TorHS) and caches (for descriptors/consensus) in the application database Once all of that will be possible, we'll be able to make a 100% clean Tor integration into GlobaLeaks (that's undergoing an architecture refactor to have a master/slave process). -- Fabio Pietrosanti (naif) HERMES - Center for Transparency and Digital Human Rights http://logioshermes.org - https://globaleaks.org - https://tor2web.org - https://ahmia.fi ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Sat, 14 Feb 2015 13:46:04 -0800 Damian Johnson ata...@torproject.org wrote: One gotcha to think about is that ADD_EPH_HS is using a variable number of positional arguments. This will limit your ability to expand this command in the future with new arguments. Also, I'd suggest renaming addrPort to addrTarget (to avoid making this sound restricted to a simple port). Hmm ok. Judging by the feedback, I'm thinking the following: * Auth is still a low-ish priority, I want to get the basic ephemeral stuff done first, and I need to read up more on how it works, and how the code is structured, before I can promise things in this area. * People seem to be ok with the tying ephemeral HSes to the originating control port (and if the only major argument against is it's a bit weird, relative to other things, well, eph. HSes are weird, and this solves a bunch of problems). So this will be implemented as planned. * ADD_EPH_HS syntax changed to something like: ADD_EPH_HS keytype:keyblob 1*(SP Port= virtPort , addrTarget) CRLF So, basically, space separated instances of Port=virtPort,addrTarget entries, of which there must be at least one. This clearly indicates the argument type and should be future proof (and also has the side benefit of being easier for me to validate. Thoughts? -- Yawning Angel pgp7Nxg_N4dT5.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] RFC: Ephemeral Hidden Services via the Control Port
Hi, The Warning: DO NOT USE MY BRANCH YET, IT HAS HAD MINIMAL TESTING AND REVIEW. IT IS NOT DONE. IT WILL BROADCAST YOUR SECRETS TO THE NSA'S ORBITAL SPACE STATION. Trac Ticket: https://trac.torproject.org/projects/tor/ticket/6411 Branch: https://github.com/Yawning/tor/compare/feature6411 With that out of the way, I've been poking at implementing a way to add hidden services at runtime via the control port. I'm calling this ephemeral hidden services because the HS state is not written out to disk, and they disappear when the tor damon is shutdown. Yes, this means that if you run kittensomgmewmew.onion and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites, you could run your tor instance entirely in a ram disk, and load the HS key manually each time from a USB token you wear around your neck. What I've come up with looks like this: ADD_EPH_HS - Adds a new ephemeral hidden service. The syntax is: ADD_EPH_HS keytype:keyblob SP 1*(port SP addrPort) CRLF keytype = RSA1024, NEW keyblob = opaque ascii serialized key material, or an algorithm specifier, in the case of keytype == NEW. port = the VIRTPORT addrPort = the TARGET A hidden service is created using the key and list of port/targets, that will persist till configuration reload or the termination of the tor process. The currently supported keytypes and keyblobs are: RSA1024 - The keyblob contains the Base64 encoding of the PKCS#1 layout DER representation of a 1024 bit RSA private key. NEW - The tor instance should generate a key with the keytype specified in the keyblob parameter. A type of BEST will generate a key using the best supported algorithm. Returns: 250-ServiceID=serviceID - The hidden service's onion address without the .onion suffix. 250-PrivateKey=keytype:keyblob - The newly generated hidden service's private key, if a keytype of NEW was specified. 250 OK - Everything went well. Proper key management, is left as an exercise for the application invoking the controller command. DEL_EPH_HS - Removes an existing ephemeral hidden service. The syntax is: DEL_EPH_HS serviceID CRLF serviceID = the hidden service identifier, without the trailing .onion suffix The hidden service specified by the given service ID is removed, and the intro points are closed. Existing connections are left as is, and it is up to the application to close them if it wishes to do so. Examples: * Adding a new ephemeral hidden service, with a pre-generated RSA key: ADD_EPH_HS RSA1024:[Base64 blob] 80 127.0.0.1:8080 250-ServiceID=blahblahblahblah 250 OK Hidden service is reachable at blahblahblahblah.onion:80 * Adding a new ephemeral hidden service, tor generates a new RSA key: ADD_EPH_HS NEW:RSA1024 22 127.0.0.1:22 250-ServiceID=hogehogehogehoge 250-PrivateKey=RSA1024:[Base64 blob] 250 OK Hidden service is reachable at hogehogehogehoge.onion:22. At a later date, the value passed back as PrivateKey can be used to re-create the hidden service. * Add a new ephemeral hidden service, tor picks the algorithm: ADD_EPH_HS NEW:BEST 1245 192.168.1.5:23 250-ServiceID=hemohemohemohemo 250-PrivateKey=Ed25519:[Base64 blob] 250 OK Hidden service is reachable at hemohemohemohemo:23. (nb: Ed25519 key support is not actually implemented yet, and is used as an example of a better algorithm.) * Remove the hidden service configured in the first example: DEL_EPH_HS blahblahblahblah 250 OK Note: ADD_EPH_HS can take more than one VIRTPORT/TARGET pair at once, the examples only show one pair for brevity. This does not support `HiddenServiceAuthorizeClient` yet in any way (neither basic nor stealth). It may in the future, but coming up with a sane interface for it is way more pain than I am willing to self-inflict at the moment. Design question that I want feedback on, primarily from control port library maintainers, and authors of applications that use the control port: * meejah suggested that ephemeral hidden services should have their lifetime tied to the originating control port connection. I think this is a good idea, but this would be the only control port command that does this sort of thing. My motivations for strongly considering implementing this is that, I don't think that adding a command to enumerate hidden services is a good idea (SketchORChat has no business in seeing what hidden services JankORFileSharing is using), and that this will automate cleanup in the event of an application crashing, or not removing it's hidden services before terminating. * Does this interface expose enough functionality, in a way
Re: [tor-dev] RFC: Ephemeral Hidden Services via the Control Port
On Fri, Feb 13, 2015, at 07:45 PM, Yawning Angel wrote: Yes, this means that if you run kittensomgmewmew.onion and are scared of the NSA's persistent attempts to extract your hidden service key, via ultrasonic laser beamed from their satellites, you could run your tor instance entirely in a ram disk, and load the HS key manually each time from a USB token you wear around your neck. A very practical use of this in the Orbot context, is that we can now store all HS identity data in an IOCipher encrypted volume, which the user can unlock with a strong passphrase when they want to start up their onionsites. Currently, all HS data is stored in the standard Tor data paths, only protected by the per-app user permissions on Android. This means the data can be accessed by rootkit capable malware apps and forensic extraction tools. With IOCipher, that would make that a great deal harder, and impossible if they were in a locked state (i.e. the key is not in memory). We are working on adding OnionShare-capabilities to Orbot, so this is well timed! +n ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
