Re: tor hidden service endpoint designs

2014-05-15 Thread David Stainton
 I also don't mind doing a 0.9.x release with this in it as a preview
 feature, even if it's not completely-fully-baked -- especially if having
 a version of this on PyPI helps your Tahoe dev. Is this something you
 need?

Thanks! That's very nice of you to offer swift code merges.

The Tahoe-LAFS Tor integration is a rather large project...
and I will be thinking about these code deployment issues soon when I finish
the server side port of Foolscap to twisted endpoints.

I'm not sure what the txsocksx author thinks about my pull request:
https://github.com/habnabit/txsocksx/pull/8

It's been a whole week without word... so I'll e-mail him soon and ask
what's up.
Maybe he'd appreciate me writing endpoint parsers for all his socks endpoints...


Thanks!
David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev


tor hidden service endpoint designs

2014-05-11 Thread David Stainton
Dear Meejah,

I'm replying by e-mail because it might be better than a conversation
inside a github pull request.

Excellent idea about not launching tor all the time! We were just
talking about that. My friend Leif Ryge and I were discussing possible
designs for this tor hidden service endpoint... and what we realized
is that the Tahoe-LAFS python process will have many endpoints it will
listen on and many endpoints it will connect to. In this case we
really want these features:

- Each Twisted process (in this case Tahoe-LAFS) would use 1 (one)
instance of Tor for all it's client and server communication. We don't
want txsocksx to have to be dependent on txtorcon... however the
txsocksx client tor endpoint could *optionally* read a txtorcon module
variable specifying the tor socks port. If that fails then the tor
client endpoint would try to find the system tor socks port.

- The first time a python process calls the `listen` method on a
hidden service endpoint, the endpoint object will either use the
system tor control port OR it will launch it's own tor process.
(great... because we can reuse your new code for this...) Which ever
it chooses it will then record this information in txtorcon module
variables... so that if the same python process later creates more
hidden service endpoints they will all use the tor process that was
chosen by the first `listen` method call. This last part about
tracking a single tor proc is not implemented yet... but Leif seemed
close to finishing a patch for that. Once this is done then I will
make the necessary changes to the txsocksx tor client endpoint.

With this design we would not really need to specify the control port.
We should probably still have it as an option.. however it isn't needed because
if we try a few ports where the system control port might be and we
cannot connect to it then in that case we should start tor with it's
control port set to an unused port.

By the way... a bit about design patterns I'm using with twisted endpoints:
In Foolscap and Tahoe-LAFS we cannot special case any usage of
endpoint objects. Foolscap currently treats all endpoint objects in
the exact same way... and in this sense is completely decoupled from
the underlying wire protocol; Foolscap is totally decoupled from Tor.
Tahoe-LAFS will have interesting Tor-friendly features like converting
non-tor endpoint descriptor strings into tor endpoint descriptor
strings... when running in Tor-only mode for instance. But
Tahoe-LAFS in this case still should be totally decoupled from
launching Tor AND decoupled from the twisted endpoints api; that is
the job of Foolscap. However... Tahoe-LAFS only knows about endpoint
descriptor strings... and this makes it's job very simple. Each
component in the system has it's own simple responsibilities... this
is why I wrote the parsers and why I cannot use extra additional
public methods calling into an endpoint object.


Sincerely,

David
___
tahoe-dev mailing list
tahoe-dev@tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev


Re: tor hidden service endpoint designs

2014-05-11 Thread David Stainton
Hi Meejah,

Briefly changing subject to the IListeningPort interface:

http://foolscap.lothar.com/trac/ticket/203

please read trac comment #19 ---
http://foolscap.lothar.com/trac/ticket/203#comment:19

Reading Foolscap trac ticket 203 caused me to think about the exact
object implementing IListeningPort interface which is fired by the Tor
Hidden Service endpoint `listen` method. A common use case for TCP
servers is to create an endpoint object with port=0; which means
randomly assigned port. Then when IListeningPort is fired you grab
your host-port tuple from the object implementing IAddress returned
from the getHost method of the object implementing IListeningPort. You
may want to pass this information to other clients in the network so
that they can connect to your server. In my opinion the Tor Hidden
Service endpoint should not return the TCP specific IAddress object...
but instead it's own.. with only the needed information (onion and
port).

The twisted documentation specifies that the getHost method of the
IListeningPort should return an IAddress object. If we take a close
look at how various twisted endpoints work... we see that they have
their own objects which implement IListeningPort and IAddress where
appropriate. I'm working on a simple changeset which will cause the
tor hidden service endpoint `listen` method to fire a Tor Onion
address specific object which implements the IListeningPort
interface... and whose getHost method will return a proper Tor Onion
address specific object which implements the IAddress interface.

My unfinished branch implementing these interfaces is here:
https://github.com/david415/txtorcon/tree/endpoint_parser_plugin-rewrite3_tor_onion_address

 Now I'm a little confused: doesn't the txsocksx client-endpoint do TCP
 over Tor connections? That is, client-side connections? It shouldn't
 really care about which Tor instance it's using (except, of course, that
 you don't want to launch lots of them), right?
 I guess maybe you mean you want launch if 0 tors, re-use if 1 logic
 here too?

No... not exactly.

In my previous e-mail I was slightly wrong and unclear when I
mentioned a couple of things...
1. A Tahoe-LAFS introducer or storage node does listen on multiple
ports... but only for onion-grids it will only need one Tor Hidden
Service onion address. Maybe it will be desirable to have a second
onion for the diagnostic web port listener...
2. I did not mean to imply that the txsocksx tor client endpoint
should launch tor. It should not. What it currently does is try a list
of local tcp ports that tor might have it's socks port listening. 9050
is the default tor socks port and 9150 the tor browser bundle default
socks port:
socks_ports_to_try = [9050, 9150]

What I would like to do is have the txsocksx tor client endpoint
optionally (if txtorcon is installed) check for a
txtorcon-launched-tor socks port and add it to the begining of
`socks_ports_to_try`... so that it tries the txtorcon tor socks port
first. If fail then retry with the next port in the list. Repeat until
`socks_ports_to_try` list is consumed.

This should be good enough. Users (developers using txsocksx and
txtorcon) have 3 options:
1. launch their own tor proc using your launch_tor api call directly
2. launch tor by calling `listen` method on a Tor Hidden Service endpoint object
3. there system tor will be used if listening on default ports

In cases 1 and 2 the launching of tor could cause a txtorcon module
attribute to record the socks port and control port. These port
numbers can then be used later by txsocksx tor client endpoints and
txtorcon tor hidden service endpoints. Some applications may want many
outgoing tor connections, some may want many hidden services and some
may want both. In this way we can accommodate all these uses
efficiently.

 One thing I need to bear in mind a little as the library author is
 that I know some people will want a private tor no matter what API
 (which is of course still possible; just use txtorcon.launch_tor).

Yes... you are right that it should have this option.

 Perhaps providing a txtorcon API for tell me all the tors you have
 launched so far would be a good thing. This would also be useful for
 use-cases where you launch lots of tor instances (e.g. things like
 Chutney, which launched a tor test network).

Yeah I agree that txtorcon should accommodate this use-case.
Let's try to clearly work out a good way to do this while still
implementing the above mentioned logic for also ensuring we have only
one instance of tor per python process. I think one instance of Tor
per python process should be the simple default way...

 I think we've already rejected overloading controlPort= for two
 different use-cases, right?

Yeah... Tahoe-LAFS and Foolscap will never need to mess with the controlPort.
Most uses of the endpoint class will not ever need to set the control
port... but someone out there
might want the ability to set it... so we should give it to them but