[tor-dev] Towards a new version of the PT spec...
So, we currently have a Pluggable Transport (PT) spec, and it kind-of sort-of works (The documentation is a mess that I'm working on cleaning up, but it's an orthogonal issue for how well it works). There are a number of problems with the current PT spec that require breaking backward compatibility to fix, so eventually I would like to do so. I'm soliciting input on what people would also like to see in a (currently hypothetical) PT spec 2.0 beyond what I already have in mind: MUST haves: * Support dual stack Bridges correctly (Multiple server endpoints per transport) * Increase the argument space beyond 510 bytes (Prop. #227). * Mandatory ExtORPort support (currently optional, but metrics are good). * Centralized logging by the calling process (Probably via stderr). * AF_UNIX support where sensible for better sandboxing. MIGHT haves: * Rename the env vars to not start with "TOR_PT". Some people claim that this is a good idea (I think it is stupid and cosmetic). * Ability to force at least clients to stop network activity without tearing the PT down. * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. UNLIKELY: * Specify an interface for where fork()/exec() isn't possible (iOS). I don't think this is makes sense because it is probably too platform/caller specific. * Allow operating both as a client and a server simultaneously. I don't see a problem with running 2 copies of something for this use case. I probably missed some things. If people have strong opinions about this, do reply, otherwise I *will* design something that I like, which will not include what other people want. Regards, -- Yawning Angel pgpk_4mdbY6Vi.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
Yawning Angel writes: > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > MUST haves: > * Support dual stack Bridges correctly (Multiple server endpoints per > transport) > * Increase the argument space beyond 510 bytes (Prop. #227). > * Mandatory ExtORPort support (currently optional, but metrics are > good). > * Centralized logging by the calling process (Probably via stderr). > * AF_UNIX support where sensible for better sandboxing. > > MIGHT haves: > * Rename the env vars to not start with "TOR_PT". Some people claim > that this is a good idea (I think it is stupid and cosmetic). I feel OK with renaming env vars to start with "PT" instead of "TOR_PT", if that will make the spec more welcoming to third-party projects > * Ability to force at least clients to stop network activity without > tearing the PT down. > * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. > > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). > I don't think this is makes sense because it is probably too > platform/caller specific. > * Allow operating both as a client and a server simultaneously. I > don't see a problem with running 2 copies of something for this > use case. > > I probably missed some things. If people have strong opinions about > this, do reply, otherwise I *will* design something that I like, which > will not include what other people want. > Hm. I think another feature that the PT land really wanted in the past was to be able to rate limit pluggable transports. I guess people would still appreciate this. I remember we tried to do something like that with prop196 but I don't remember if we subsequently decided that it was ridiculous and/or stupid. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
> On 8 Sep 2015, at 08:45, Yawning Angel wrote: > > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > ... > > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). >I don't think this is makes sense because it is probably too >platform/caller specific. I imagine that this would require a PT-as-thread(s) interface, which is out of scope, as iOS is a single platform. It seems to me that using a PT on iOS could be done in a similar way to using tor. (That is, if you can’t fork tor from an iOS app, then forking PTs is the least of your worries.) I’m hoping someone has developed a generic way of doing this, at least for network apps. What are the ChatSecure people doing for their XMMP + Tor chat accounts? Launching pthreads? Tim (teor) Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP: 968F094B (ABFED1AC & A39A9058 expire 15 Sep 2015) teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F (From 1 Sep 2015) signature.asc Description: Message signed with OpenPGP using GPGMail ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
On Tue, 8 Sep 2015 17:39:58 +1000 Tim Wilson-Brown - teor wrote: > > > On 8 Sep 2015, at 08:45, Yawning Angel > > wrote: > > > > So, we currently have a Pluggable Transport (PT) spec, and it > > kind-of sort-of works (The documentation is a mess that I'm working > > on cleaning up, but it's an orthogonal issue for how well it works). > > > > There are a number of problems with the current PT spec that require > > breaking backward compatibility to fix, so eventually I would like > > to do so. > > > > I'm soliciting input on what people would also like to see in a > > (currently hypothetical) PT spec 2.0 beyond what I already have in > > mind: > > > > ... > > > > UNLIKELY: > > * Specify an interface for where fork()/exec() isn't possible > > (iOS). I don't think this is makes sense because it is probably too > >platform/caller specific. > > I imagine that this would require a PT-as-thread(s) interface, which > is out of scope, as iOS is a single platform. It seems to me that > using a PT on iOS could be done in a similar way to using tor. (That > is, if you can’t fork tor from an iOS app, then forking PTs is the > least of your worries.) I was thinking PTs as a shared object of some sort with presumably but not necessarily lots of threads under the hood. But I marked this as UNLIKELY precisely because it's a lot of effort for a single platform (however important) that isn't targeted by an official Tor anything at this point. It also is a platform I don't have access to, and never will, so unless someone is willing to chime in on how to properly support this sort of thing in a way that doesn't clutter up the PT spec and implementations with a lot of extra stuff it won't happen. > I’m hoping someone has developed a generic way of doing this, at > least for network apps. What are the ChatSecure people doing for > their XMMP + Tor chat accounts? Launching pthreads? Don't know, don't care at this moment since it's totally orthogonal to drafting a new Pluggable Transports spec (That doesn't mean that it isn't important, just trying to keep discussion on track.). Regards, -- Yawning Angel pgptXr0MPflc5.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
It is my understanding that a sponsored project is coming up to work a pluggable transport 2.0 specification and implementation. I've also heard that the first step for this is to have a meeting where we get together with people that either use pluggable transports or implement them, for the purpose of soliciting feedback on the existing specification and gathering design requirements for 2.0. So perhaps the drafting of a new specification should be deferred until this is organized. Although of course any feedback gathered in this email thread can also be presented at the meeting. On Mon, Sep 7, 2015 at 6:45 PM, Yawning Angel wrote: > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > MUST haves: > * Support dual stack Bridges correctly (Multiple server endpoints per > transport) > * Increase the argument space beyond 510 bytes (Prop. #227). > * Mandatory ExtORPort support (currently optional, but metrics are > good). > * Centralized logging by the calling process (Probably via stderr). > * AF_UNIX support where sensible for better sandboxing. > > MIGHT haves: > * Rename the env vars to not start with "TOR_PT". Some people claim > that this is a good idea (I think it is stupid and cosmetic). > * Ability to force at least clients to stop network activity without > tearing the PT down. > * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. > > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). > I don't think this is makes sense because it is probably too > platform/caller specific. > * Allow operating both as a client and a server simultaneously. I > don't see a problem with running 2 copies of something for this > use case. > > I probably missed some things. If people have strong opinions about > this, do reply, otherwise I *will* design something that I like, which > will not include what other people want. > > Regards, > > -- > Yawning Angel > > ___ > tor-dev mailing list > tor-dev@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev > > ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
On Mon, 14 Sep 2015 16:14:41 -0400 Brandon Wiley wrote: > It is my understanding that a sponsored project is coming up to work a > pluggable transport 2.0 specification and implementation. I've also > heard that the first step for this is to have a meeting where we get > together with people that either use pluggable transports or > implement them, for the purpose of soliciting feedback on the > existing specification and gathering design requirements for 2.0. So > perhaps the drafting of a new specification should be deferred until > this is organized. Although of course any feedback gathered in this > email thread can also be presented at the meeting. This would be the first I heard of either. Till someone tells me details about this, I think the sensible thing to do would be to discuss it here... It's not as if lots of people seem interested in giving feedback anyway at present so I'll probably end up drafting something that does what I want it to do (which fixes the major current shortcomings of the interface from my POV), at which point all the other people will appear to complain. Regards, -- Yawning Angel pgpcmJoDQOZYh.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
Yawning Angel transcribed 3.3K bytes: > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > MUST haves: > * Support dual stack Bridges correctly (Multiple server endpoints per > transport) Do you mean in terms of running the same transport on say 10 IP addresses (#7884), or just dual stack support (#11211)? Or… both? :) I would really like both. :D > * Increase the argument space beyond 510 bytes (Prop. #227). > * Mandatory ExtORPort support (currently optional, but metrics are > good). > * Centralized logging by the calling process (Probably via stderr). > * AF_UNIX support where sensible for better sandboxing. > > MIGHT haves: > * Rename the env vars to not start with "TOR_PT". Some people claim > that this is a good idea (I think it is stupid and cosmetic). It's probably a good idea to shorten it to "PT_" as asn suggested, but removing prefixes altogether (as you obviously know) is believed to possibly result in envvar collision (perhaps this concern has since become archaic). > * Ability to force at least clients to stop network activity without > tearing the PT down. > * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. All of the above seem like good ideas to add. I think I also agree that the following seems out-of-scope (and likely for Apple to change the rules/APIs out from under us). > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). > I don't think this is makes sense because it is probably too > platform/caller specific. > * Allow operating both as a client and a server simultaneously. I > don't see a problem with running 2 copies of something for this > use case. As always, I'm glad to provide help with this stuff, whether spec or code changes, if you want it. -- ♥Ⓐ isis agora lovecruft _ OpenPGP: 4096R/0A6A58A14B5946ABDE18E207A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt signature.asc Description: Digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
At Psiphon we often discuss (and get asked about) using Tor's pluggable transports directly. The cost/benefit balance hasn't yet been in favour of doing this, but if there's discussion of a big PT revamp... maybe Psiphon should indicate how the cost side of the balance could come down for us. We're obviously not a priority for what Tor does with PTs, but there's surely no harm in us mentioning our wishlist. So... What would be best for us is if PTs were written in Go and implemented the net.Conn[1] interface. We have had good results with the composability of net.Conn implementations: an obfuscated SSH net.Conn on top of a meek net.Conn[2] on top of a upstream proxy net.Conn[3] on top of a TCPConn net.Conn[4]. Layering in a PT net.Conn would be sane and clean and reasonably easy. Conversely: Python is difficult-to-impossible due to runtime distribution. Separate executables are difficult-to-impossible due to Android PIE requirements and distribution size bloat. Anyway, if this is of any interest we can discuss it further. (Note: Probably Lantern people are reading this too. And probably they would benefit from the same things we would, since their architecture is similar to ours.) [1]: https://golang.org/pkg/net/#Conn [2]: https://github.com/Psiphon-Labs/psiphon-tunnel-core/blob/master/psiphon/meekConn.go [3]: https://github.com/Psiphon-Labs/psiphon-tunnel-core/tree/master/psiphon/upstreamproxy [4]: https://golang.org/pkg/net/#TCPConn Adam Pritchard Psiphon Inc. > Date: Mon, 7 Sep 2015 22:45:52 + > From: Yawning Angel > To: tor-dev@lists.torproject.org > Subject: [tor-dev] Towards a new version of the PT spec... > Message-ID: <20150907224552.6959b...@schwanenlied.me> > Content-Type: text/plain; charset="us-ascii" > > So, we currently have a Pluggable Transport (PT) spec, and it kind-of > sort-of works (The documentation is a mess that I'm working on > cleaning up, but it's an orthogonal issue for how well it works). > > There are a number of problems with the current PT spec that require > breaking backward compatibility to fix, so eventually I would like to > do so. > > I'm soliciting input on what people would also like to see in a > (currently hypothetical) PT spec 2.0 beyond what I already have in mind: > > MUST haves: > * Support dual stack Bridges correctly (Multiple server endpoints per > transport) > * Increase the argument space beyond 510 bytes (Prop. #227). > * Mandatory ExtORPort support (currently optional, but metrics are > good). > * Centralized logging by the calling process (Probably via stderr). > * AF_UNIX support where sensible for better sandboxing. > > MIGHT haves: > * Rename the env vars to not start with "TOR_PT". Some people claim > that this is a good idea (I think it is stupid and cosmetic). > * Ability to force at least clients to stop network activity without > tearing the PT down. > * Deprecate SOCKS4a, and make SOCKS5 mandatory for clients. > > UNLIKELY: > * Specify an interface for where fork()/exec() isn't possible (iOS). > I don't think this is makes sense because it is probably too > platform/caller specific. > * Allow operating both as a client and a server simultaneously. I > don't see a problem with running 2 copies of something for this > use case. > > I probably missed some things. If people have strong opinions about > this, do reply, otherwise I *will* design something that I like, which > will not include what other people want. > > Regards, > > -- > Yawning Angel -- Adam Pritchard Psiphon Inc. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
On Thu, 17 Sep 2015 14:28:24 -0400 Adam Pritchard wrote: > At Psiphon we often discuss (and get asked about) using Tor's > pluggable transports directly. The cost/benefit balance hasn't yet > been in favour of doing this, but if there's discussion of a big PT > revamp... maybe Psiphon should indicate how the cost side of the > balance could come down for us. > > We're obviously not a priority for what Tor does with PTs, but there's > surely no harm in us mentioning our wishlist. So... > > What would be best for us is if PTs were written in Go and implemented > the net.Conn[1] interface. We have had good results with the > composability of net.Conn implementations: an obfuscated SSH net.Conn > on top of a meek net.Conn[2] on top of a upstream proxy net.Conn[3] on > top of a TCPConn net.Conn[4]. Layering in a PT net.Conn would be sane > and clean and reasonably easy. > > Conversely: Python is difficult-to-impossible due to runtime > distribution. Separate executables are difficult-to-impossible due to > Android PIE requirements and distribution size bloat. The Guardian project people seem to manage to handle distributing multiple separate executables (tor, meek, obfs4proxy). I do not know what they do, maybe they just take the size hit. > Anyway, if this is of any interest we can discuss it further. I'm interested in hearing what people want but: * I personally am against forcing any particular language on people, since the point of PTs is that they are easy for interested parties to write. * Pluggable Transport implementations sharing address space with the Tor binary is basically never going to happen due to security concerns (iOS may do weird things, but iOS is not an officially supported target). The revamp is more of a "keeping the current model the same (ie: fork/exec a helper process with a known external interface), how can we make said external interface better" than "change the entire Pluggable Transport concept". > (Note: Probably Lantern people are reading this too. And probably they > would benefit from the same things we would, since their architecture > is similar to ours.) Now despite all of this, the obfs2/3/4 and ScrambleSuit implementations I did for obfs4proxy do in fact implement a net.Conn interface[0] and always have[1]. Regards, -- Yawning Angel [0]: https://gitweb.torproject.org/pluggable-transports/obfs4.git/tree/transports/base/base.go [1]: I reserve the right to implement a hypothetical `obfs5` in something that is not Go, depending on the availability of required libraries/primitives, my mood, the phase of the moon, and planetary alignment. pgpwFNWoJiMAq.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Towards a new version of the PT spec...
I mostly just want to second everything Adam P has already said and to emphasize some details. For us, PTs implemented in Go is a huge win and is enough that we'll likely integrate obfs4 directly into Lantern (https://www.getlantern.org). I happen to have just been looking at the net.Conn interface implementation in particular, which I agree is very handy. The runtime size issue is really a significant one. Particularly in Iran, a download size over maybe 10MB is just often a deal breaker in practice. More broadly it would be amazing on our end if PTs were a little more decoupled (or pluggable!) from not just Tor but also SOCKS, HTTP, etc, and when I say that I mean at all of the following layers: 1) Code 2) Configuration 3) Documentation For example, just a simple example of wrapping an existing net.Conn with obfs4 without getting into the details of the Tor config and environment variables or even jumping through SOCKs hoops would likely have seen obfs4 integrated into Lantern *today* and deployed at scale potentially next week. In that example, I think making PTs more easily usable even outside of the whole PT framework would also be a huge improvement - i.e. for us we'd ideally not take the performance hit of jumping through an extra proxy even though that approach makes a lot of sense in the larger PT context. All of that is doable now of course, but takes a little digging. Now, to be clear, the beauty of PTs, and what makes this discussion even possible, is how decoupled they already are. I suggest that taking that decoupling to the next level could have really nice cascading effects for PTs. For example, I think it would make both us and Psiphon quite likely to use them (TBH we'll probably use them again anyway, as I said), but that also means two strong decent-sized full time teams contributing to debugging, patching, documentation, etc. We also both run at significant scale in censored regions where PTs are obviously particularly useful, so we'd be likely to contribute to things like benchmarking and performance improvements if not additional PTs themselves. Anyway, thanks for listening. I think PTs are a truly awesome example of a cool "architecture of participation", and we'd love to help make them even more participatory and powerful! -Adam Fisk Brave New Software Project, Inc. On Thu, Sep 17, 2015 at 11:28 AM, Adam Pritchard wrote: > At Psiphon we often discuss (and get asked about) using Tor's > pluggable transports directly. The cost/benefit balance hasn't yet > been in favour of doing this, but if there's discussion of a big PT > revamp... maybe Psiphon should indicate how the cost side of the > balance could come down for us. > > We're obviously not a priority for what Tor does with PTs, but there's > surely no harm in us mentioning our wishlist. So... > > What would be best for us is if PTs were written in Go and implemented > the net.Conn[1] interface. We have had good results with the > composability of net.Conn implementations: an obfuscated SSH net.Conn > on top of a meek net.Conn[2] on top of a upstream proxy net.Conn[3] on > top of a TCPConn net.Conn[4]. Layering in a PT net.Conn would be sane > and clean and reasonably easy. > > Conversely: Python is difficult-to-impossible due to runtime > distribution. Separate executables are difficult-to-impossible due to > Android PIE requirements and distribution size bloat. > > Anyway, if this is of any interest we can discuss it further. > > (Note: Probably Lantern people are reading this too. And probably they > would benefit from the same things we would, since their architecture > is similar to ours.) > > [1]: https://golang.org/pkg/net/#Conn > [2]: > https://github.com/Psiphon-Labs/psiphon-tunnel-core/blob/master/psiphon/meekConn.go > [3]: > https://github.com/Psiphon-Labs/psiphon-tunnel-core/tree/master/psiphon/upstreamproxy > [4]: https://golang.org/pkg/net/#TCPConn > > Adam Pritchard > Psiphon Inc. > > >> Date: Mon, 7 Sep 2015 22:45:52 + >> From: Yawning Angel >> To: tor-dev@lists.torproject.org >> Subject: [tor-dev] Towards a new version of the PT spec... >> Message-ID: <20150907224552.6959b...@schwanenlied.me> >> Content-Type: text/plain; charset="us-ascii" >> >> So, we currently have a Pluggable Transport (PT) spec, and it kind-of >> sort-of works (The documentation is a mess that I'm working on >> cleaning up, but it's an orthogonal issue for how well it works). >> >> There are a number of problems with the current PT spec that require >> breaking backward compatibility to fix, so eventually I would like to >> do so. >> >> I'm soliciting input on what people would also like to see in a >> (currently hypothe
