Re: MinGW Target Re-Enabled on TaskCluster

[Georg Koppen]

Tom Ritter:
> Previous Thread:
> https://groups.google.com/forum/#!topic/mozilla.dev.platform/r3mYWbb42pM
> 
> As of a few hours ago, there is a new Tier 2 MinGW build on TaskCluster.
> It's in the 'Windows MinGW all' line, with the group WMC64 for 'Windows
> MinGW Clang x64'.
>

Yay, thanks so much for that. I am pretty sure it will help us saving a
ton of work once we plan to switch to the mingw-w64/clang toolchain, as
your work on integrating the mingw-w64/gcc toolchain into TaskCluster
already did.

> 
> The MinGW builds are part of the Tor Uplift project, where we work closely
> with Tor to upstream their patches and in general move Tor Browser closer
> and closer to 'Firefox with Addons and Pref Changes' - instead of a fork of
> Firefox with a bunch of custom patches.
> 
> Tor is currently using ESR releases: the bump to ESR60 (which occurred last
> week! It's a huge release for them with a lot of UX improvements:
> https://blog.torproject.org/new-release-tor-browser-80 ) was a lot smoother
> - build-wise - than other bumps because we had the MinGW build running in
> TC. Without keeping the MinGW build working, every ESR they have to go
> through a potentially colossal amount of work on a short timescale to get
> the build working under MinGW again. By minimizing their effort in fixing
> the build, rebasing patches and the like - we can free up their limited
> resources to continue to research and experiment on privacy technology like
> First Party Isolation and Fingerprinting Resistance.

Let me jump in here as the one responsible for coordinating our efforts
to switch Tor Browser to ESR 60, and the one working on the
mingw-w64-based toolchain upgrades in the past. Having to deal with the
mingw-w64 upgrade for a new ESR has been one of the top three pain
points in previous years.

I usually started weeks before we began with the "official" transition
part, to estimate how much is broken and how complex fixes would be by
bisecting and filing bugs at bugzilla. Thanks to the help of Jacek Caban
we always had the toolchain ready before we needed to switch, but it was
pretty close sometimes. All this work has essentially not been needed
anymore this time, thanks to the effort of Tom and others, and we are
very grateful for that.

Given that background I am more than happy to see the results of the
continued work in that direction with the arrival of the
mingw-w64/clang-based Windows build on Taskcluster.

Thanks Mozilla and thanks to those engineers in particular who helped
(and still help) with that effort.

Georg



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Georg Koppen]

Hi!

Tom Ritter:
> As part of our work with Tor, we’ve been working on getting a MinGW-based
> build of Windows into TaskCluster. Tor is currently using ESR releases, and
> every ESR they have to go through a large amount of work to get the build
> working under MinGW again; by continually building (and testing) that build
> we’ll be able to cut weeks to months of effort for them each ESR release.
> (Not breaking the MinGW build is also a necessity if they were ever to move
> off ESRs.)
> 
> Unlike our normal Windows builds with msvc or clang-cl, this is a
> cross-compile on Linux for Windows, as well as the first time in a long
> time we’ve built for Windows with gcc. Building with gcc for Windows has
> exposed a number of new warnings
>  to clean up, as well
> as some C++ spec incompatibilities that MSVC (and clang-cl) accept. All of
> the breaks have been resolved; and a lot of the warnings and similar are
> either resolved or in-progress.
> 
> Effective this weekend, MinGW is on Treeherder, with the single build
> target win32-mingw32 (debug). Its toolchain builds are under TMW (with a
> few under TL because they are generic Linux packages.)
> 
> The one-click loaner system works for MinGW, and we’re going to allow this
> to bake at Tier 2 for a while. Your help in keeping the build green is
> greatly appreciated, and if you hit a MinGW bug you can’t decipher, I’ll be
> happy to help, just send me an email or a needinfo. Some of the most common
> things that cause errors for MinGW:
> 
> 
>  - #include  instead of #include  and similar
> 
>  - _uuidof() instead of IID_* ref
> 
> 
>  - Casting nullptr to bool or int
> 
>  - Using the (in)correct assembly code
> 
>  - Breaking --disable-webrtc, --disable-sandbox, or --disable-accessibility
> 
>  - MinGW lagging behind on defining new constants Microsoft has defined
> 
> There are a few things left outstanding
> 
> for the MinGW build, including webrtc
> , stylo
>  (which may be the
> most difficult in this list), the sandbox
> ,
> --enable-accessibility
> , cleaning
> up warnings , and 
> running
> tests . In the
> future, there is hope (by me at least) to throw all of this work away.
> Compiling on Linux for Windows with clang (or clang-cl) would probably be a
> significant improvement for Tor’s builds, and would enable them to take
> advantage of CFI and SEH, as well as simplifying the number of platforms
> Mozilla supports. Chrome has been pushing towards this
> , so we’re
> keeping an eye on their progress.
> 
> Finally, I want to thank everyone who brought this long process to this
> point: everyone who wrote patches, explained things (sometimes over and
> over again), and reviewed. That’s including but not limited to: boklm,
> dmajor, froydnj, glandium, georg, jacek, jrmuizel, and ted; plus the dozen
> or so more people who have reviewed my patches all over the codebase.

That's great news, Tom! Thanks for working on this project and all its
numerous child bugs, greatly appreciated. I know from experience how
time-consuming and tedious this is.

To stress a point you made in your email above: your efforts and those
of other Mozilla engineers that helped with this project does indeed
save us weeks of work when preparing major Tor Browser releases based on
a new ESR. We can now use this valuable time to fix more privacy issues
faster which benefits both Tor Browser and Firefox users alike because
we upstream those patches as we think not only Tor users should benefit
from them.

As to your future plans, yes, we are happy if we can help with the
effort to move away from using GCC for Windows cross-builds and move to
clang instead. That's not only due to the SEH and CFI support you
mentioned. Rather, it would allow us to get rid of annoying GCC bugs,
like https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384, as well and
would significantly simpify our cross-compile setups we need to maintain.

Thanks,
Georg



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reproducible builds

[Georg Koppen]

Hi,

David Bruant:

[snip]

> Out of curiosity, how has is the TOR team handled points 1 and 2?

1) We remove the .chk files before generating the final package.
2) We have deterministic tar/zip wrappers we deploy, e.g.:

https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/build-helpers/dtar.sh

Georg

[snip]



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform