Re: [tor-dev] Preferred compression type?
Thanks, Nick! On 9/13/19 4:24 PM, Nick Mathewson wrote: > On Fri, Sep 13, 2019 at 2:05 PM Steve Snyder wrote: >> >> Given the multiple compression types supported (none, lzma, zlib, zstd), >> what is the order of preference for runtime use? >> >> Put another way, which compression method(s) should be supported to get >> optimal runtime performance from a Tor node? > > For big objects like consensuses or consensus diffs that are sent over > and over, relays prefer to use whichever compression method has the > highest compression -- that's lzma2, then zstd, then zlib, then none. > Lzma2 (aka xz) is more expensive to calculate, but the relays only > need to calculate it once per compressed object, and then they can > send it over and over. > > For smaller objects that are compressed in a stream (descriptors and > microdescriptors), relays will not use xz, since it would be to > expensive to recompute it for every stream. They'll prefer zstd, then > zlib, then none. > > So if you want to save bandwidth above all, you should enable all > compression algorithms. > > If you want to save CPU above all, you should enable all compression > algorithms except xz. > > If you want to save bandwidth and CPU, I _think_ that enabling all the > compression algorithms will result in Tor making good choices (as > described above). But I'd appreciate benchmarks if anybody has tried > it both ways to find out. > > cheers, > ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Preferred compression type?
On Fri, Sep 13, 2019 at 2:05 PM Steve Snyder wrote: > > Given the multiple compression types supported (none, lzma, zlib, zstd), > what is the order of preference for runtime use? > > Put another way, which compression method(s) should be supported to get > optimal runtime performance from a Tor node? For big objects like consensuses or consensus diffs that are sent over and over, relays prefer to use whichever compression method has the highest compression -- that's lzma2, then zstd, then zlib, then none. Lzma2 (aka xz) is more expensive to calculate, but the relays only need to calculate it once per compressed object, and then they can send it over and over. For smaller objects that are compressed in a stream (descriptors and microdescriptors), relays will not use xz, since it would be to expensive to recompute it for every stream. They'll prefer zstd, then zlib, then none. So if you want to save bandwidth above all, you should enable all compression algorithms. If you want to save CPU above all, you should enable all compression algorithms except xz. If you want to save bandwidth and CPU, I _think_ that enabling all the compression algorithms will result in Tor making good choices (as described above). But I'd appreciate benchmarks if anybody has tried it both ways to find out. cheers, -- Nick ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] Preferred compression type?
Given the multiple compression types supported (none, lzma, zlib, zstd), what is the order of preference for runtime use? Put another way, which compression method(s) should be supported to get optimal runtime performance from a Tor node? Thanks. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
Hi all, Just saw this thread while heading out the door and wanted to mention that we already have a reproducible build setup for Tor and obfs4proxy binaries for Android and Linux. The binaries are published on JCenter. Hans-Christoph, hope this shortens your path! :-) https://code.briarproject.org/briar/tor-reproducer https://code.briarproject.org/briar/go-reproducer https://bintray.com/briarproject/org.briarproject Cheers, Michael On 13/09/2019 16:24, Santiago Torres-Arias wrote: > On Fri, Sep 13, 2019 at 12:32:06PM +0200, Nicolas Vigier wrote: >> On Thu, 12 Sep 2019, Hans-Christoph Steiner wrote: >> >>> >>> And third, and tips on getting a Linux shared library to build >>> reproducibly. E.g. is faketime a hard requirement? >> >> Usually it's not needed to use faketime. It's only useful if the >> toolchain has bugs that cannot easily be fixed, causing some timestamps >> to be inserted somewhere. >> > +1, using faketime so liberally would just mean there are other, bigger > underlying issues (e.g., lack of support for S_D_E...) > > Cheers! > -Santiago. > > > ___ > tor-dev mailing list > tor-dev@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev > 0x11044FD19FC527CC.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
On Fri, Sep 13, 2019 at 12:32:06PM +0200, Nicolas Vigier wrote: > On Thu, 12 Sep 2019, Hans-Christoph Steiner wrote: > > > > > And third, and tips on getting a Linux shared library to build > > reproducibly. E.g. is faketime a hard requirement? > > Usually it's not needed to use faketime. It's only useful if the > toolchain has bugs that cannot easily be fixed, causing some timestamps > to be inserted somewhere. > +1, using faketime so liberally would just mean there are other, bigger underlying issues (e.g., lack of support for S_D_E...) Cheers! -Santiago. signature.asc Description: PGP signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
On Thu, Sep 12, 2019 at 10:50 AM Hans-Christoph Steiner wrote: > > > Hey all, > > I'm currently working on tor for Android as part of a Guardian Project > project. One key goal is making a shareable, reproducible build process > for the tor daemon for Android. Then this would be published to > MavenCentral as an Android AAR package to be used in all the apps that > include tor (Tor Browser, Orbot, Briar, Thali, etc). I have cleaned up > the existing build process a lot, so now I'm down to troubleshooting > reproducible issues. So we're clear on the expectations and goals of this, you're working on compiling Tor reproducibly without using rbm, correct? And are you imagining the Guardian Project would build and upload these packages to MavenCentral, as part of your work? I wonder if extracting tor binaries and shared libraries from tor-browser-build (or another project on top of rbm) is an easier route, in the longer term. I don't know what issue you'll find when you begin diving into the differences between builds (if there are any, at all). Overall, I think this is a great plan and I'd like to see more applications bundled with Tor (and CalyxOS is exciting). However, as Georg already mentioned, this may not be something that fits into Tor Browser in the future. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
On Fri, Sep 13, 2019 at 7:51 AM Hans-Christoph Steiner wrote: > > > > teor: > > Hi, > > > >> On 12 Sep 2019, at 20:50, Hans-Christoph Steiner > >> wrote: > >> > >> Then that work > >> will hopefully be extended into sharing tor between apps, e.g. letting > >> Briar, Tor Browser, etc share the tor SOCKS proxy to other apps that > >> want to use it. That would happen via Android mechanisms like Intents to > >> manage the discovery of SOCKS ports. > > > > It's not always safe to have apps share Tor: a malicious website in one app > > can use various caches to discover activity in other apps. And there may > > be similar data leaks in other shared data structures or network > > connections. > > > > How do these data leaks affect your use cases? > > Is there some documentation of these leaks somewhere so I can dive into > it? Like what kind of caches? Browser caches? We're mostly talking > about apps that are not browsers, like messaging, nextcloud, etc. I have a half-baked (and half-written) proposal for this. I'll try to finish it within the next few weeks. The summary of it is that in an ideal situation every application should run its own tor client. If two applications share a tor client, and they both use the control port, then they both learn information about connections requested by the other - maybe that is onion service addresses only one app should use, or it leaks user behavior. Even in the case where multiple apps don't have control port access, they can potentially use the same circuits if their proxy settings are misconfigured (which leaks timing information locally and possibly leaks information about applications and the user on the network). Instead of every tor client wasting mobile data with duplicate downloads, I'm imagining having a Tor gateway (for lack of a better word). This essentially is a local Tor bridge relay that provides a single point of entry into and exit from the device. This solves most of the problems and it can be setup today except this completely breaks the assumptions within Tor that a Bridge is an external first-hop. In a configuration like this, the Tor gateway is a local hop, so it reduces a three-hop circuit to two-hops. And, to make matters worse, every second-hop would continue to be selected randomly, so the connection into the Tor network would be a random node and not a Guard. And, third, a client loses the ability to configure an actual bridge (with or without a pluggable transport) for its connection into the network. With this being said, something like Bridge Guards [0] would solve these problems. Unfortunately, Bridge Guards are still an open question. I was hoping to look into this in the near future. This setup would probably require some synchronization between tor node if, for example, multiple applications should use the same onion service, but this can happen outside of tor. [0] https://gitweb.torproject.org/torspec.git/tree/proposals/188-bridge-guards.txt ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
On Thu, 12 Sep 2019, Hans-Christoph Steiner wrote: > > And third, and tips on getting a Linux shared library to build > reproducibly. E.g. is faketime a hard requirement? Usually it's not needed to use faketime. It's only useful if the toolchain has bugs that cannot easily be fixed, causing some timestamps to be inserted somewhere. signature.asc Description: PGP signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] New Orbot, tor-android and AndroidPluggableTransport updates
Nathan Freitas:
> A new Orbot is out, with a bug fix related to obfs4proxy installation,
> and a new tor!
Good stuff! Is it intended that I only see an x86_64, x86, and arm64-v8a
version but no armv7 one available? It seems suddenly Orbot is not
compatible anymore with my device (and I suspect a bunch of other users
have a similar problem)
Georg
> We've also updated the tor-android library, with the latest 0.4.1.5
> release, and all is working well.
>
> https://github.com/guardianproject/tor-android/releases/tag/tor-android-binary-tor-0.4.1.5
>
> and we updated our AndroidPluggableTransports library, which was at the
> root of our "Orbot Bridge" bug, not properly installing the binary from
> the APK bundle.
>
> https://github.com/guardianproject/AndroidPluggableTransports/releases/tag/1.0.7
>
> These libraries and versions are available via our Guardian Project
> gradle/maven repo hosted on Github:
>
> maven { url
> "https://raw.githubusercontent.com/guardianproject/gpmaven/master"; }
>
> implementation 'org.torproject:tor-android-binary:0.4.1.5-rc'
>
> implementation 'info.pluggabletransports.aptds:apt-dispatch-library:1.0.7'
> implementation 'info.pluggabletransports.aptds:apt-meek-obfs4-legacy:1.0.7'
>
> Everything is building again in our CI pipeline here:
> https://gitlab.com/guardianproject/orbot/pipelines
> and we nearly have automated on-device testing working via the cloud
> services Bitbar and/or Browserstack.
>
> Life is good, again.
>
> +n
>
>
> Forwarded Message
> Subject: Orbot 16.1.2 RC-2 with tor-0.4.1.5-rc
> Date: Tue, 10 Sep 2019 00:42:56 -0400
> From: Nathan of Guardian
> Organisation: Guardian Project
> To: Guardian Dev
>
>
>
> Orbot 16.1.2-RC-2-tor-0.4.1.5-rc
> https://github.com/guardianproject/orbot/releases/tag/16.1.2-RC-2-tor-0.4.1.5-rc
>
> * Found and fixed bug related to bridges not working
> (https://github.com/guardianproject/orbot/issues/242)
> * Updated tor to 0.4.1.5
>
> ARM64 APK:
> https://github.com/guardianproject/orbot/releases/download/16.1.2-RC-2-tor-0.4.1.5-rc/Orbot-16.1.2-RC-2-tor-0.4.1.5-rc-fullperm-arm64-v8a-release.apk
> ARMEABI APK:
> https://github.com/guardianproject/orbot/releases/download/16.1.2-RC-2-tor-0.4.1.5-rc/Orbot-16.1.2-RC-2-tor-0.4.1.5-rc-fullperm-armeabi-v7a-release.apk
>
> @n8fr8 n8fr8 released this 5 minutes ago ยท 0 commits to master since
> this release
>
> c2e679d (HEAD -> master, tag: 16.1.2-RC-2-tor-0.4.1.5-rc) update to
> 16.1.2-RC-2-tor-0.4.1.5-rc
> 1d45a3d update APT Pluggable Transport library to 1.0.7 - this should
> fix one issue with installing the correct ofbs4proxy binary - also small
> changes to ensure no lock states and reduce log noise
> efec61d (tag: 16.1.2-RC-1-tor-0.4.0.4-rc, newport/master) update to
> 16.1.2-RC-1-tor-0.4.0.4-rc
> 237b388 yet another attempt at fixing the phantom zulu locale bug!
> a9efc3c disable tests for now in CI build
> 918bcca (origin/master, origin/HEAD) remove unused layouts for app-mini
> 08b670f update paths for app-mini code
> cefac58 putting classes back into main app package, and not "mini"
> 97d2aac improvements for bridge handling for #242
> 0e40c07 update badvpn tun2socks library
> 034844a remove fileprovider from app-mini since it is used just for HS
> d95f197 make keystore props reading not fail if not present
> d0640c6 remove output from repo
> e8f3158 remove binaries from repo
> 5845a4c update build tools to 29
> 003d9ea update build tools to 29
> 9c6de06 update build tools to 29
> a266c65 add new functionality to allow for checking connections against
> app blacklist
> d98e5fb update changelog for 16.1.1-BETA-2-tor-0.4.0.4-rc-orbotservice
> Assets 6
> Orbot-16.1.2-RC-2-tor-0.4.1.5-rc-fullperm-arm64-v8a-release.apk 10.3 MB
> Orbot-16.1.2-RC-2-tor-0.4.1.5-rc-fullperm-arm64-v8a-release.apk.asc 833
> Bytes
> Orbot-16.1.2-RC-2-tor-0.4.1.5-rc-fullperm-armeabi-v7a-release.apk 9.94 MB
> Orbot-16.1.2-RC-2-tor-0.4.1.5-rc-fullperm-armeabi-v7a-release.apk.asc
> 833 Bytes
>
>
>
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
signature.asc
Description: OpenPGP digital signature
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
Hans-Christoph Steiner: > Georg Koppen: >> Hans-Christoph Steiner: >>> >>> Hey all, >>> >>> I'm currently working on tor for Android as part of a Guardian Project >>> project. One key goal is making a shareable, reproducible build process >>> for the tor daemon for Android. Then this would be published to >>> MavenCentral as an Android AAR package to be used in all the apps that >>> include tor (Tor Browser, Orbot, Briar, Thali, etc). I have cleaned up >>> the existing build process a lot, so now I'm down to troubleshooting >>> reproducible issues. >>> >>> First off, can anyone see any objections to switching Tor Browser, >>> Orbot, Briar, etc. to use GPG-signed reproducible binaries via >>> MavenCentral for the tor dameon? >> >> We want to include building tor and all its dependencies in >> tor-browser-build/rbm to have the latest tor for Android in our nightly >> builds and respective alpha and stable versions in our alpha and stable >> browsers. We have a ticket for that for a while now in our bug tracker >> but did not get to it so far.[1] The plan is to pick that work up in >> November after Tor Browser 9 is out. >> >> As to whether other projects would be interested in that, dunno. But I >> guess some at least would? >> >> Georg >> >> [1] The parent ticket for that work is: >> https://trac.torproject.org/projects/tor/ticket/28704. > > If building tor+libevent+openssl+libz+liblzma for Android was done > reproducibly and shipped via MavenCentral, would you consider using it? > Seems like we'd want this tor binary to be synced to the Tor Browser > version requirements anyway, since that's the "standard configuration". What about our nightly build requirement? Oh, and to complicate that: we build tor nightlies with Rust enabled to be able to test Rust code. And would do so for Android, too. And to further complicate matters: we plan to switch to NSS to test that part of tor in a Tor Browser context as well. (It's been long on the agenda but I finally want to get to that after Tor Browser 9 is out) And then there has been times where we actually needed to ship tor patches ourselves because they were not merged/released yet (although, luckily that's been a while ago). There might be need for such an option in the future, too. So, all in all I am skeptical that Tor Browser fits into your plans. Georg signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] reproducible builds for Android tor daemon
teor: > Hi, > >> On 12 Sep 2019, at 20:50, Hans-Christoph Steiner >> wrote: >> >> Then that work >> will hopefully be extended into sharing tor between apps, e.g. letting >> Briar, Tor Browser, etc share the tor SOCKS proxy to other apps that >> want to use it. That would happen via Android mechanisms like Intents to >> manage the discovery of SOCKS ports. > > It's not always safe to have apps share Tor: a malicious website in one app > can use various caches to discover activity in other apps. And there may > be similar data leaks in other shared data structures or network > connections. > > How do these data leaks affect your use cases? Is there some documentation of these leaks somewhere so I can dive into it? Like what kind of caches? Browser caches? We're mostly talking about apps that are not browsers, like messaging, nextcloud, etc. Currently, running multiple tor daemons is really a showstopper for most mobile users in the world, both because of battery usage and bandwidth costs. I guess there was some progress towards getting tor sleeping more effectively as well as not consuming as much bandwidth in the background. So the big question is: will it be feasible to have a usable tor on mobile that aggressively sleeps to stop consuming any battery and bandwidth when not directly in use? If so, then running multiple tor daemons should be workable. If not, then we need to find a way to share the tor daemon across all apps in the device. With Orbot, all apps are already sharing one tor daemon, so this isn't a new development. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556 ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
