Re: Linux distro readiness for Rust in Gecko

[natanael . copa]

On Tuesday, March 22, 2016 at 11:06:18 PM UTC, Petr Cerny wrote:
> i.stakenvic...@gmail.com wrote:
> > Technically speaking, as i've been told at least, rust can still be
> > built from scratch if ocaml is available on the platform, starting
> > with rust-0.6, to 0.7, 0.8, 0.9, 1.0 and then finally 1.7 or newer.
> > If a distro really does ban cross-compiling to jump directly to
> > rust-1.7 or newer then their developers can stoll go this route to
> > package it.  Theoretically it only needs to be done once, right?
> 
> Question is, whether we'll be fine with some 7 iterations like you 
> suggest or we'd need the ~300 steps Henri mentioned.

I´d like to build rust for alpine linux, which uses musl libc. There is big 
interest among alpine users to get rust working for alpine.

I´d say that 7 iteration is too much, because it is 7 x number of 
architectures. In alpine case it is currently 3 arch, x86, x86_64 and arm. It 
will likely be more.

It would be *very* helpful to be able to bootstrap rust directly from 
C/python/perl/whatever.

-nc
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[aturon]

On Thursday, March 24, 2016 at 10:21:12 AM UTC-7, Chris Peterson wrote:
> On 3/24/16 10:13 AM, atu...@mozilla.com wrote:
> > The Rust core team met yesterday and developed a complete plan for Rust to 
> > be build with the previous stable compiler as snapshot. We will be landing 
> > the infrastructure needed to do this shortly.
> >
> > In terms of release timing, it will take some time for this change to fully 
> > propagate. Our most recent Rust snapshot happened after the 1.8 beta, 
> > leading to the following picture:
> >
> > - Rust 1.8 -- already in beta; hits stable on Apr 14th
> > - Ad hoc snapshot
> > - Rust 1.9 -- builds from ad hoc snapshot
> > - Rust 1.10 -- builds from Rust 1.9 stable
> >
> > The Rust 1.10 stable release will happen about 15 weeks from now, and under 
> > this plan will be the first release to bootstrap this way.
> 
> Aaron, so going forward, Rust version N should always be buildable by 
> version N-1 (starting with N = 1.10)?

Correct.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Chris Peterson]

On 3/24/16 10:13 AM, atu...@mozilla.com wrote:

The Rust core team met yesterday and developed a complete plan for Rust to be 
build with the previous stable compiler as snapshot. We will be landing the 
infrastructure needed to do this shortly.

In terms of release timing, it will take some time for this change to fully 
propagate. Our most recent Rust snapshot happened after the 1.8 beta, leading 
to the following picture:

- Rust 1.8 -- already in beta; hits stable on Apr 14th
- Ad hoc snapshot
- Rust 1.9 -- builds from ad hoc snapshot
- Rust 1.10 -- builds from Rust 1.9 stable

The Rust 1.10 stable release will happen about 15 weeks from now, and under 
this plan will be the first release to bootstrap this way.


Aaron, so going forward, Rust version N should always be buildable by 
version N-1 (starting with N = 1.10)?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[aturon]

On Tuesday, March 22, 2016 at 4:25:35 PM UTC-7, atu...@mozilla.com wrote:
> The Rust compiler has had over 300 snapshots, so re-bootstrapping from ocaml 
> would take at least 900 compiles (each one requires three stages of internal 
> builds); by my estimation, this would take at least a month of constant 
> compilation. Moreover, it's not clear this is even possible: a few points 
> along the early history involved manual edits as well.
> 
> I don't think this line of investigation is worth pursuing. I think the 
> reasonable route is to:
> 
> - Get Rust snapshotting based on the previous stable compiler, for ease of 
> following the release train. (I hope to have news on that front tomorrow.)
> 
> - Seed distros from a relatively recent point in the chain.

An update on the snapshot front:

The Rust core team met yesterday and developed a complete plan for Rust to be 
build with the previous stable compiler as snapshot. We will be landing the 
infrastructure needed to do this shortly. 

In terms of release timing, it will take some time for this change to fully 
propagate. Our most recent Rust snapshot happened after the 1.8 beta, leading 
to the following picture:

- Rust 1.8 -- already in beta; hits stable on Apr 14th
- Ad hoc snapshot
- Rust 1.9 -- builds from ad hoc snapshot
- Rust 1.10 -- builds from Rust 1.9 stable

The Rust 1.10 stable release will happen about 15 weeks from now, and under 
this plan will be the first release to bootstrap this way.

In terms of bootstrapping the distros themselves, as I mentioned before the 
simplest approach seems to be starting with a recent Rust snapshot and going 
from there.

How does all this sound?

Aaron
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Selena Deckelmann]

Hi!

We are holding off on packaging with nix for a couple quarters because of
how much work in the TC-worker we have ahead of us.

Let's have a nix conversation the first week of the quarter and then we can
see what low hanging fruit there is. Right now, lots of people are stressed
with gsoc and outreachy and end of quarter goal work :)

I'm so glad you're advocating for this :). Have you talked with gps at all
about nix and reproducible builds?

-Selena

On Thu, Mar 24, 2016 at 4:50 AM Petr Cerny  wrote:

> Henri Sivonen wrote:
> > On Wed, Mar 23, 2016 at 1:51 AM, Petr Cerny  wrote:
> >> Saying "Linux is good, we can actually do things there, that are
> >> more complicated elsewhere, but you are hindering our development"
> >>  sound a bit unfair.
> >
> > There is no contradiction between Rust working the best on Linux and
> > being of the opinion that limiting ourselves to Rust features that
> > were part of rustc when Debian stable last shipped is not okay.
> > Clearly, there are distros that are don't have a policy issue with
> > Firefox gaining a dependency on an actively-developed compiler. It's
> > just that the need of a policy exception is what generates email.
>
> Right, but Debian is part of the ecosystem. I think you might have
> (unpleasant) reminiscences about the later days of WinXP, but people
> were using it for some reason, and I bet they have good reasons to use
> Debian stable and not some faster moving distributions.
>
> > It's hard for me to tell if you represent a distro with which a
> > rustc dependency will be a non-issue, since your comments run the
> > gamut of implying your distro had a compiler bootstrapping policy
> > stricter than Fedora's to saying that six weeks is viable.
>
> Because it all depends on precise conditions. (And my comments were
> meant also more generally than just for SLE). Six weeks is viable
> technically iff all works as expected. Like Sylvester mentioned, once
> there is a hiccup (rustc pulling in an update of llvm, llvm pulling in
> update of gcc) we're all out of luck.
>
> The dependency on non-stable build tools is not just a technical problem
> - anyone not having a special taste for using bleeding edge technologies
> will tell you such approach is broken. It may help the development big
> time, but for packagers it is just pain.
>
> >> The all-from-source, stable interfaces and similar policies are
> >> there for a reason.
> >
> > Again, "stable" is a loaded word. Maintaining interface
> > compatibility in the sense that stuff written to the old interface
> > doesn't break on update is reasonable to want, but e.g. in the case
> > of Debian that's not the criterion for getting to update a package
> > (absent policy exception). rustc maintains compatibility in the sense
> > that source code that compiled with some version (1.0 or later) of
> > rustc continues to compile with later versions.
>
> Which is good, but only half of the problem. The other is that the
> language itself is not stable enough to provide some sort compatibility
> in the other direction. Which is why (with my limited understanding of
> Debian policies) assume it could live in Unstable or Testing but not
> Stable.
>
> > As I understand Debian policy, even if upstream says that it's their
> > explicit goal to maintain compatibility in the sense that new
> > versions don't break stuff written for old versions, Debian assumes
> > that upstream won't meet such a goal and instead only cherry-picking
> > security fixes from the upstream is permitted (absent the sort of
> > exception Chromium got).
>
> Well... most of the time things work. The problem are the cases when
> seemingly innocent upstream changes break working setups (because
> upstream doesn't care about history, they want progress). Policies like
> this are motivated exactly by this sort of things happening in the past.
> Does Chromium pull in new versions of compilers or comparable packages
> every other month?
>
> >> Actually you could package Firefox for generic Linux by building
> >> your own GCC and bundling all libraries (the whole GTK stack) into
> >>  the tarball/rpm - at some point replicating good portion of
> >> packages installed on an average desktop system. ... You don't
> >> want to go there.
> >
> > Mozilla already ships binary tarballs compiled with a GCC version of
> > Mozilla's choice. Those tarballs don't even need to include GTK+,
> > because except for the changes between gtk2 and gtk3, GTK+ does a
> > good job maintaining ABI compatibility.
>
> Let's just say that we do have separate GTK package for Firefox on older
> SLE code streams. Although "older" in this case is not the 3-5 years you
> would see with Debian. I think you understand my feelings the day it
> turns out I'll have to package and regularly update LLVM and rustc every
> 6 weeks...
>
> >> Distributions should be fine with 6 weeks, if changes are announced
> >> with reasonable time buffer.
> >
> > I'd 

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

Henri Sivonen wrote:

On Wed, Mar 23, 2016 at 1:51 AM, Petr Cerny  wrote:

Saying "Linux is good, we can actually do things there, that are
more complicated elsewhere, but you are hindering our development"
 sound a bit unfair.


There is no contradiction between Rust working the best on Linux and
being of the opinion that limiting ourselves to Rust features that
were part of rustc when Debian stable last shipped is not okay.
Clearly, there are distros that are don't have a policy issue with
Firefox gaining a dependency on an actively-developed compiler. It's
just that the need of a policy exception is what generates email.


Right, but Debian is part of the ecosystem. I think you might have 
(unpleasant) reminiscences about the later days of WinXP, but people 
were using it for some reason, and I bet they have good reasons to use 
Debian stable and not some faster moving distributions.



It's hard for me to tell if you represent a distro with which a
rustc dependency will be a non-issue, since your comments run the
gamut of implying your distro had a compiler bootstrapping policy
stricter than Fedora's to saying that six weeks is viable.


Because it all depends on precise conditions. (And my comments were
meant also more generally than just for SLE). Six weeks is viable
technically iff all works as expected. Like Sylvester mentioned, once
there is a hiccup (rustc pulling in an update of llvm, llvm pulling in
update of gcc) we're all out of luck.

The dependency on non-stable build tools is not just a technical problem
- anyone not having a special taste for using bleeding edge technologies
will tell you such approach is broken. It may help the development big
time, but for packagers it is just pain.


The all-from-source, stable interfaces and similar policies are
there for a reason.


Again, "stable" is a loaded word. Maintaining interface
compatibility in the sense that stuff written to the old interface
doesn't break on update is reasonable to want, but e.g. in the case
of Debian that's not the criterion for getting to update a package
(absent policy exception). rustc maintains compatibility in the sense
that source code that compiled with some version (1.0 or later) of
rustc continues to compile with later versions.


Which is good, but only half of the problem. The other is that the
language itself is not stable enough to provide some sort compatibility
in the other direction. Which is why (with my limited understanding of
Debian policies) assume it could live in Unstable or Testing but not Stable.


As I understand Debian policy, even if upstream says that it's their
explicit goal to maintain compatibility in the sense that new
versions don't break stuff written for old versions, Debian assumes
that upstream won't meet such a goal and instead only cherry-picking
security fixes from the upstream is permitted (absent the sort of
exception Chromium got).


Well... most of the time things work. The problem are the cases when
seemingly innocent upstream changes break working setups (because
upstream doesn't care about history, they want progress). Policies like
this are motivated exactly by this sort of things happening in the past.
Does Chromium pull in new versions of compilers or comparable packages
every other month?


Actually you could package Firefox for generic Linux by building
your own GCC and bundling all libraries (the whole GTK stack) into
 the tarball/rpm - at some point replicating good portion of
packages installed on an average desktop system. ... You don't
want to go there.


Mozilla already ships binary tarballs compiled with a GCC version of
Mozilla's choice. Those tarballs don't even need to include GTK+,
because except for the changes between gtk2 and gtk3, GTK+ does a
good job maintaining ABI compatibility.


Let's just say that we do have separate GTK package for Firefox on older
SLE code streams. Although "older" in this case is not the 3-5 years you
would see with Debian. I think you understand my feelings the day it 
turns out I'll have to package and regularly update LLVM and rustc every 
6 weeks...



Distributions should be fine with 6 weeks, if changes are announced
with reasonable time buffer.


I'd expect it to make sense to update rustc every six weeks without
specific notice. Maybe Firefox won't need a new rustc for every
Firefox release, but considering the pace of development of rustc
and the Rust standard library, I think it would make the most sense
to plan with the assumption that both Firefox and rustc update every
six weeks.


OK, but 6 weeks for a compiler update is not big-enough time buffer: if
at some point rustc indeed pulls in newer LLVM things will get nasty.
You can't update system LLVM, so you have to build a separate one. If
that one for some reason requires newer GCC... well you get the picture.

If you can guarantee such things won't happen, good. Telling packagers 6
weeks ahead that for Firefox update they will need newer LLVM and

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Wed, Mar 23, 2016 at 1:51 AM, Petr Cerny  wrote:
> Henri Sivonen wrote:
> Well... changes can be done, but have to be announced. Which is why this
> thread is a Good Thing (TM).

That's why I started this thread.

> By the way, from the picture referenced above I think that in the layout
> engines part of the jungle you (as in Firefox/Gecko) sadly are the smaller
> animal for quite some time now.

That's why it's so important that we be able to leverage Rust to our advantage.

>> But consider: Rust as a technology works the best on Linux. (IIRC,
>> debugging info works better on Linux than on Mac and it's pretty
>> clear that Linux and Mac are better matches for Rust development
>> than Windows.) It's very messed up that the tech that works the best
>> on Linux is the hardest to ship to users on Linux.
>
>
> Saying "Linux is good, we can actually do things there, that are more
> complicated elsewhere, but you are hindering our development" sound a
> bit unfair.

There is no contradiction between Rust working the best on Linux and
being of the opinion that limiting ourselves to Rust features that
were part of rustc when Debian stable last shipped is not okay.
Clearly, there are distros that are don't have a policy issue with
Firefox gaining a dependency on an actively-developed compiler. It's
just that the need of a policy exception is what generates email.

It's hard for me to tell if you represent a distro with which a rustc
dependency will be a non-issue, since your comments run the gamut of
implying your distro had a compiler bootstrapping policy stricter than
Fedora's to saying that six weeks is viable.

> The all-from-source, stable
> interfaces and similar policies are there for a reason.

Again, "stable" is a loaded word. Maintaining interface compatibility
in the sense that stuff written to the old interface doesn't break on
update is reasonable to want, but e.g. in the case of Debian that's
not the criterion for getting to update a package (absent policy
exception). rustc maintains compatibility in the sense that source
code that compiled with some version (1.0 or later) of rustc continues
to compile with later versions. As I understand Debian policy, even if
upstream says that it's their explicit goal to maintain compatibility
in the sense that new versions don't break stuff written for old
versions, Debian assumes that upstream won't meet such a goal and
instead only cherry-picking security fixes from the upstream is
permitted (absent the sort of exception Chromium got).

> Actually you could package Firefox for generic Linux by building your
> own GCC and bundling all libraries (the whole GTK stack) into the
> tarball/rpm - at some point replicating good portion of packages
> installed on an average desktop system. These would be used by firefox
> only and would (mostly) work (mostly) everywhere. You would actually be
> doing exactly what we do for old code streams, only across multiple
> distributions. You don't want to go there.

Mozilla already ships binary tarballs compiled with a GCC version of
Mozilla's choice. Those tarballs don't even need to include GTK+,
because except for the changes between gtk2 and gtk3, GTK+ does a good
job maintaining ABI compatibility.

>>> Yet back to the point: you certainly can change whichever tool you
>>>  need or want, but the timing is crucial. Stable distribution
>>> maintainers are used to these things happening every once in a
>>> while, so if you announce it loudly a couple of releases ahead, it
>>>  will be perfectly fine and nobody is going to complain
>>
>>
>> Not even Debian?
>
>
> Your call, Mike... :)
>
> Genrally: it can be done

Great!

> Distributions should be fine with 6 weeks, if changes are announced with
> reasonable time buffer.

I'd expect it to make sense to update rustc every six weeks without
specific notice. Maybe Firefox won't need a new rustc for every
Firefox release, but considering the pace of development of rustc and
the Rust standard library, I think it would make the most sense to
plan with the assumption that both Firefox and rustc update every six
weeks.

> So 6 weeks
> clearly is viable now.

Great!

> Having say rustc-X and rustc-X+4 for FF-Y and FF-Y+7 (which is the ESR
> stepping), shouldn't be such a problem either

It makes sense to assume that if Firefox Y depends on rustc X, Firefox
Y+7 will depend on rustc X+7.

> (although it does add
> quite some work especially of you can't build rustc-X+4 with rustc-X
> directly)

But if you also ship rapid-release Firefox in addition to ESR, surely
you'll need the rustc versions between X and X+7 for that. Since rustc
continues to compile old programs, you could build Firefox Y.0.Z ESR
with rustc X+Z, so you shouldn't need to keep an older rustc around
for Firefox ESR while you are updating rustc to compile non-ESR
Firefox.

> Stable for LTS/enterprise means: doesn't change existing interfaces and
> functionality. As in: f(x) is always be available, does 

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

Henri Sivonen wrote:

The comments on
https://bugzilla.mozilla.org/show_bug.cgi?id=1175546 suggest that
Mozilla deferred relying on GCC bug fixes until after 45 ESR in
order for Debian not to have to backport a compiler for
oldstable. Is that the case? That is, is Debian's policy already
hindering Mozilla's ability to avoid C++ compiler bugs that the
compiler upstream has fixed?


... Longer version:

It definitely is not just Debian. With all due respect, Mozilla
doesn't rule an isolated island - like all others it shares a
jungle with other animals of varying sizes, shapes and habits.


Yes, but the Linux jungle depends on the outside for the browser
engines (Gecko, Blink and WebKit) that it relies on and the numbers
that Web developers look to assess the relevance of these engines
(which translates into whether sites are made in a way that these
engines work for Linux users) happens outside the Linux jungle.


Hm... I don't have numbers, but if I'm to believe
https://en.wikipedia.org/wiki/Comparison_of_web_browser_engines#/media/File:Usage_share_of_web_browsers_(Source_StatCounter).svg 
,

I don't like what I see. I guess no one here does. But should it become
too cumbersome to package Firefox at some point something would have to
give. Let's not go there and just say I really do not want to change my
email signature.


It's not reasonable for a part of the already small jungle to be
entitled to limit the Gecko development inside and outside the jungle
to old compilers.


Well... changes can be done, but have to be announced. Which is why this
thread is a Good Thing (TM).

By the way, from the picture referenced above I think that in the layout
engines part of the jungle you (as in Firefox/Gecko) sadly are the 
smaller animal for quite some time now. ESR is imho one of the (arguably 
not so many) big features that are slowing down further drop.



But consider: Rust as a technology works the best on Linux. (IIRC,
debugging info works better on Linux than on Mac and it's pretty
clear that Linux and Mac are better matches for Rust development
than Windows.) It's very messed up that the tech that works the best
on Linux is the hardest to ship to users on Linux.


Saying "Linux is good, we can actually do things there, that are more
complicated elsewhere, but you are hindering our development" sound a
bit unfair. "Rust works best on Linux," you say. So that part of the
jungle that is putting some restrictions in place is actually not that
bad, right? Yet these are just parts of the overall picture and you
can't really have one without the other.

My feeling is, that you might not be fully aware that it's not just the
technology side one needs to be aware of. The all-from-source, stable
interfaces and similar policies are there for a reason. From my point of
view part of the problem are the rapid changes in rustc, requiring very
recent rustc for building itself (that is my take-away from this thread
so far; more on that below).

Without the LTS/Enterprise distributions Linux would be less usable,
because there would be much smaller installed base and thus less people
working on it (and consequently probably also less pressure on quality).
I'm not saying by how much, but expect such a difference to be measurable.


I'm quite sure there are problems with other compilers/systems than
just GCC/Linux, but due to their larger target user deployment and
distribution model, they are being worked around. I know it is
different (in *many* aspects), but just imagine you had to solve a
similar issue with Microsoft or Apple (i.e. them distributing
Firefox and strongly preferring to build it with what they use for
 building the rest of their systems with).


I don't think a comparison with Microsoft and Apple is in the
distros' favor. Microsoft's undocumented ABI has the practical effect
of locking us to their C++ compiler, but they don't make us use just
the C++ features and compiler bugs that were available when Windows
XP shipped.


Yes, but just like GCC has bugs, MSVC has them too. That you may not be
affected doesn't make it a better development environment.


Apple promotes clang and we are at liberty to use clang from llvm.org
instead of Apple-shipped clang and we have the option to patch clang
if we need to.



As for imagining if Microsoft and Apple imposed requirements that
they don't: The important point is that they don't!


Exactly. And they are also not so good for playing with Rust.

Actually you could package Firefox for generic Linux by building your
own GCC and bundling all libraries (the whole GTK stack) into the
tarball/rpm - at some point replicating good portion of packages
installed on an average desktop system. These would be used by firefox
only and would (mostly) work (mostly) everywhere. You would actually be
doing exactly what we do for old code streams, only across multiple
distributions. You don't want to go there. Spending time on that while
it is re-done by distributions in parallel 

Re: Linux distro readiness for Rust in Gecko

[aturon]

On Tuesday, March 22, 2016 at 4:06:18 PM UTC-7, Petr Cerny wrote:
> i.stake...@gmail.com wrote:
> > On Thursday, March 17, 2016 at 4:31:46 PM UTC-4, Henri Sivonen
> >> If distro policy bans ongoing cross-compliation, I guess the
> >> distro would need to replicate the Rust project's compiler
> >> compilation version lineage on each architecture after
> >> bootstrapping with cross-compilation.
> >>
> >
> > Technically speaking, as i've been told at least, rust can still be
> > built from scratch if ocaml is available on the platform, starting
> > with rust-0.6, to 0.7, 0.8, 0.9, 1.0 and then finally 1.7 or newer.
> > If a distro really does ban cross-compiling to jump directly to
> > rust-1.7 or newer then their developers can stoll go this route to
> > package it.  Theoretically it only needs to be done once, right?
> 
> Question is, whether we'll be fine with some 7 iterations like you 
> suggest or we'd need the ~300 steps Henri mentioned.

The Rust compiler has had over 300 snapshots, so re-bootstrapping from ocaml 
would take at least 900 compiles (each one requires three stages of internal 
builds); by my estimation, this would take at least a month of constant 
compilation. Moreover, it's not clear this is even possible: a few points along 
the early history involved manual edits as well.

I don't think this line of investigation is worth pursuing. I think the 
reasonable route is to:

- Get Rust snapshotting based on the previous stable compiler, for ease of 
following the release train. (I hope to have news on that front tomorrow.)

- Seed distros from a relatively recent point in the chain.

I also want to echo what Ralph said: the Rust team takes backwards 
compatibility very seriously, so for ESR releases it should be quite feasible 
to use a newer version of Rust. (I'm not saying there is never breakage, but 
it's very rare and almost always such that code can easily be modified to work 
with both the old and new versions of the compiler.)

Aaron
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

i.stakenvic...@gmail.com wrote:

On Thursday, March 17, 2016 at 4:31:46 PM UTC-4, Henri Sivonen

If distro policy bans ongoing cross-compliation, I guess the
distro would need to replicate the Rust project's compiler
compilation version lineage on each architecture after
bootstrapping with cross-compilation.



Technically speaking, as i've been told at least, rust can still be
built from scratch if ocaml is available on the platform, starting
with rust-0.6, to 0.7, 0.8, 0.9, 1.0 and then finally 1.7 or newer.
If a distro really does ban cross-compiling to jump directly to
rust-1.7 or newer then their developers can stoll go this route to
package it.  Theoretically it only needs to be done once, right?


Question is, whether we'll be fine with some 7 iterations like you 
suggest or we'd need the ~300 steps Henri mentioned.


Petr
--
Petr Cerny
Mozilla/OpenSSH maintainer for SUSE Linux
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Ralph Giles]

On Tue, Mar 22, 2016 at 9:31 AM,   wrote:

> Given that I know very little about rust, and putting aside the activities of 
> Rust upstream's releases in general, what do we expect is going to be needed 
> here in terms of a rust package to build an ESR vs a regular firefox release? 
>  I assume / hope that if firefox-52ESR can be built with say, rust-2.1 , that 
> it can also be built with the rust-2.4 that's needed to build firefox-58 ??

I would expect this to be the case. So far, the rust team has done a
very good job maintaining backward compatibilty for language features
once they reach stable release.

 -r
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[i . stakenvicius]

On Friday, March 18, 2016 at 3:05:25 PM UTC-4, band...@mozilla.com wrote:
> 
> This seems like a workable solution, as horrible as it is. One could 
> similarly make distro packages of rustc specifically for building firefox 
> (called 'rustc-fx' for example), and put them on the same ESR upgrade 
> schedule as Firefox.


Please no.  It would be only a matter of time before firefox/mozilla would no 
longer be able to be built with a standard upstream rust package due to some 
sort of idiosyncracy in rust-fx.


> 
> As another idea: Rust could maintain an LTS branch designed to be upgraded in 
> conjunction with Firefox, that both keeps the language on a stable revision 
> while also exposing backported features that Firefox needs in a controlled 
> way. So Firefox would commit to being on e.g. 1.12 for 3 years, but could 
> import 'future' features. Then LTS distros would put rustc LTS point releases 
> on the same upgrade schedule as Firefox ESR upgrades.


Given that I know very little about rust, and putting aside the activities of 
Rust upstream's releases in general, what do we expect is going to be needed 
here in terms of a rust package to build an ESR vs a regular firefox release?  
I assume / hope that if firefox-52ESR can be built with say, rust-2.1 , that it 
can also be built with the rust-2.4 that's needed to build firefox-58 ??  

If that isn't the case, ie, the version of rustc/libstd/etc that's installed 
can only build source that's compatible with that version of the rust language, 
then us distros will just need to keep around (and likely allow parallel 
installation of) all the different rust versions.  Not ideal, but it isn't like 
this is anything we haven't had to deal with before.  The main issue here will 
be with backporting rust code from mozilla-central to fix the ESR's, since its 
possible the newer code is going to need a newer rust language.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[i . stakenvicius]

On Thursday, March 17, 2016 at 4:31:46 PM UTC-4, Henri Sivonen wrote:
> On Thu, Mar 17, 2016 at 3:01 PM, Martin Stransky wrote:
> > Well, what about other arches? ppc(64), s390, ai64...?
> 
> All architectures currently supported by Fedora, Debian and Ubuntu
> (but not Gentoo!) already have LLVM back ends. rustc hasn't been
> ported to all those architectures, but the level of effort required
> should be within reason considering that the LLVM back ends are there
> and rustc has already been ported to target multiple ones of the
> LLVM-supported ISAs.

Gentoo does have system LLVM support on x86{,_64} ppc{,64} arm{,64} and sparc 
-- no alpha, hppa or ia64 at of yet.  The rust situation on gentoo is still in 
flux however.  Gentoo is very much going to support cross-compilation of rust 
and rust-based projects too.

Regarding other platforms -- as of firefox-45.0 , the only platforms that work 
properly are those that have a jit, or have appropriate none-jit 
AtomicOperations.h ..  Since moz45 the only commits i've seen have been to add 
ppc/ppc64 support (in moz47), so as far as I am aware hppa, s390, ia64, etc are 
effectively not supported at runtime already (they MOZ_CRASH on startup).  
Regardless of whether or not those platforms have rust support, some work in 
js/src/jit/ is going to be required before it'll matter.


> If distro policy bans ongoing cross-compliation, I guess the distro
> would need to replicate the Rust project's compiler compilation
> version lineage on each architecture after bootstrapping with
> cross-compilation.
> 

Technically speaking, as i've been told at least, rust can still be built from 
scratch if ocaml is available on the platform, starting with rust-0.6, to 0.7, 
0.8, 0.9, 1.0 and then finally 1.7 or newer.  If a distro really does ban 
cross-compiling to jump directly to rust-1.7 or newer then their developers can 
stoll go this route to package it.  Theoretically it only needs to be done 
once, right?

Ian
-
Ian Stakenvicius
a...@gentoo.org
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

Henri Sivonen wrote:

 > Sure, once you are on similar level of stability to GCC, it is not such
 > a big problem. Note that GCC 5.3.0 (to my best knowledge) can be
 > compiled by GCC-3.4 (10 years difference). Once rustc is capable of
 > being compiled by a one or two years older version of itself, the
 > problem is pretty much solved. Even 9 months would be mostly fine.

rustc is already in openSUSE, so evidently, it's past the bootstrap
phase. Can't the enterprise version of SUSE promote a package from
openSUSE and have openSUSE (Tumbleweed at least) go through every rustc
release so as not to break the chain?


I'll have to check how those packages were built originally and discuss 
that with security team - there are differences between openSUSE and SLE.


Thanks
Kind regards
Petr

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Mar 21, 2016 7:21 PM, "Petr Cerny"  wrote:
>

> Sure, once you are on similar level of stability to GCC, it is not such
> a big problem. Note that GCC 5.3.0 (to my best knowledge) can be
> compiled by GCC-3.4 (10 years difference). Once rustc is capable of
> being compiled by a one or two years older version of itself, the
> problem is pretty much solved. Even 9 months would be mostly fine.

rustc is already in openSUSE, so evidently, it's past the bootstrap phase.
Can't the enterprise version of SUSE promote a package from openSUSE and
have openSUSE (Tumbleweed at least) go through every rustc release so as
not to break the chain?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Mon, Mar 21, 2016 at 6:49 PM,   wrote:
> I'm surprised to hear that *10.4* is still expected to work. The oldest 
> version I've heard Rust needed to support is 10.6.

10.6 on Intel is the oldest supported by Mozilla. Cameron maintains a
port to 10.4 on PPC: http://www.floodgap.com/software/tenfourfox/

-- 
Henri Sivonen
hsivo...@hsivonen.fi
https://hsivonen.fi/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

bander...@mozilla.com wrote:
>> Actually, this would be pretty much what GCC does: it bootstraps
>> itself in three phases, with just one `make` command. And although
>> the scale is much longer, compiling for example GCC-5.3.0 with GCC
> 
> This is also what rustc does - it bootstraps off of itself. rustc is
> a self-hosting systems compiler comparable to gcc. Asking rustc to
> bootstrap off of yet another compiler is just kicking the can down
> the road. Eventually Rust wants to be in the same position as gcc,
> where distros have their own binary lineage and are bootstrapping
> their own compiler.

Sure, once you are on similar level of stability to GCC, it is not such
a big problem. Note that GCC 5.3.0 (to my best knowledge) can be
compiled by GCC-3.4 (10 years difference). Once rustc is capable of
being compiled by a one or two years older version of itself, the
problem is pretty much solved. Even 9 months would be mostly fine.

> If we were to put the effort into transpiling to C (or flat out
> writing a rust compiler in C) so that it could be recompiled it would
> just be an intermediate step until distros have their own bootstrap
> working. Bootstrapping off of transpiled C is not much different than
> bootstrapping off a binary blog. The C is not going to be readable.

Ok, if we are talking about output from LLVM backend, than it is exactly
same as using a binary blob (more below). A simple Rust compiler that
would shorten the chain from point zero to today by a half wouldn't
change that much. If it cut things down to say 10 iterations, it would
help. It wouldn't need to be fast or efficient, since it's product would
only be used once.

>> The problem is having to go through some 300 iterations (even if
>> one would probably do that only once on each distribution release).
>> Which as far as I understand - and please do correct me if I'm
>> wrong - would be the case (presumably because rustc's code is
>> progressively using its own new features).
> 
> I don't think anybody is expecting this. And in any case it's so
> impracticable that it's impossible. Distros will pick some existing
> binary snapshot to trust and then bootstrap off their own binaries.

I actually tend to think you are wrong on this one.

I personally would go through the 300 iterations (even if just once)
rather than using a random binary blob. Unless it is signed with a
secure key and that person/corporation is going to take (legal)
responsibility for any security problems caused by the blob itself (i.e.
compiling in a back-door).

Now, while it's not up to me to make these decisions, I can assure you
that these matters are not taken lightly for enterprise Linux.

> From my point of view there are two options that would greatly
>> help:
>> 
>> 1) a bootstrapping rust compiler written in C  that would work in
>> a small number of rounds as has been described above. It could as
>> well be C++, Python, Perl or whatever can be commonly found in
>> distributions - the lower the better to keep the dependency tree
>> smaller.
> 
> If this tool existed it would definitely help fill in the gaps. It
> would be a great amount of work to maintain.

If it is too much of a burden, 300 iterations may suddenly seem more
appealing. Once we get to point where 2) below applies it's (almost) a
piece of cake (I actually mentioned that above already).

>> 2) a more stable rustc codebase - stable in the sense that a
>> recompile would only be necessary on the order of magnitude of ESR
>> releases (note: I'm writing this without any knowledge about
>> current rustc compile requirements). This would allow distributions
>> to perform the long iteration chain once and then update rustc only
>> from time to time.

Thanks
Kind regards
Petr
-- 
Petr Cerny
Mozilla/OpenSSH maintainer for SUSE Linux
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[banderson]

On Monday, March 21, 2016 at 9:43:24 AM UTC-7, Petr Cerny wrote:
> Chris Peterson wrote:
> > On 3/20/16 3:04 AM, Henri Sivonen wrote:
> >> On Sat, Mar 19, 2016 at 2:27 PM,   wrote:
> >>> > On Thursday, 17 March 2016 12:23:32 UTC, Henri Sivonen  wrote:
>  >> (rustc originally bootstrapped with OCaml, but
>  >> building the whole history of Rust from the OCaml days to present
>  >> every time Fedora builds Firefox seems excessively impractical.)
> >>> >
> >>> > Out of interest, would that actually involve building every single 
> >>> > Linux snapshot from 
> >>> > https://github.com/rust-lang/rust/blob/master/src/snapshots.txt in 
> >>> > sequence? All 319 of them?
> >> Presumably, yes, if you want to bootstrap from OCaml. Maybe you could
> >> skip some snapshots, but you don't know which ones in advance.
> >> Fortunately, e.g. the Fedora policy doesn't require bootstrapping all
> >> the way from OCaml.
> > 
> > How does Debian bootstrap other compilers like Go that are partially 
> > self-hosted? Would a Rust compiler written in C/C++ (e.g. generated by 
> > an LLVM back-end that emitted C/C++ code?) avoid the bootstrap policy 
> > requirements?
> 
> That would definitely help (and not just for Debian), since
> bootstrapping as far as possible is a generally preferred direction on
> Linux, with the only exception of the basic toolchain (binutils, GCC),
> libc and couple more packages that form a basic system (shell, make).
> That is: you take a minimal working system, recompile new versions of
> the packages with those from older release, then use the new minimal
> system to build anything else.
> 
> If you can get sources that will allow building current rustc within say
> 3 build cycles, i.e.:
> 
> [C sources]---(C compiler)-->[rustc-intermediate]
> 
> [Rust sources]---(rustc-intermediate)-->[rustc-current]
> 
> [Rust sources]---(rustc-current)-->[rustc-current]
> 
> it will be just fine. Obviously 2 cycles would be nicer, but this even
> with one added layer in between is manageable. If the distributed source
> package contained both the rust sources and a bootstrapping compiler and
> maybe even allowed building the final binary with a single command (a la
> `[configure script]; make`) it would behave just like any other package.
> 
> Actually, this would be pretty much what GCC does: it bootstraps itself
> in three phases, with just one `make` command. And although the scale is
> much longer, compiling for example GCC-5.3.0 with GCC

This is also what rustc does - it bootstraps off of itself. rustc is a 
self-hosting systems compiler comparable to gcc. Asking rustc to bootstrap off 
of yet another compiler is just kicking the can down the road. Eventually Rust 
wants to be in the same position as gcc, where distros have their own binary 
lineage and are bootstrapping their own compiler.

If we were to put the effort into transpiling to C (or flat out writing a rust 
compiler in C) so that it could be recompiled it would just be an intermediate 
step until distros have their own bootstrap working. Bootstrapping off of 
transpiled C is not much different than bootstrapping off a binary blog. The C 
is not going to be readable.

> 
> The problem is having to go through some 300 iterations (even if one
> would probably do that only once on each distribution release). Which as
> far as I understand - and please do correct me if I'm wrong - would be
> the case (presumably because rustc's code is progressively using its own
> new features).

I don't think anybody is expecting this. And in any case it's so impracticable 
that it's impossible. Distros will pick some existing binary snapshot to trust 
and then bootstrap off their own binaries.

> 
> From my point of view there are two options that would greatly help:
> 
> 1) a bootstrapping rust compiler written in C  that would work in a
> small number of rounds as has been described above. It could as well be
> C++, Python, Perl or whatever can be commonly found in distributions -
> the lower the better to keep the dependency tree smaller.

If this tool existed it would definitely help fill in the gaps. It would be a 
great amount of work to maintain.

> 
> 2) a more stable rustc codebase - stable in the sense that a recompile
> would only be necessary on the order of magnitude of ESR releases (note:
> I'm writing this without any knowledge about current rustc compile
> requirements). This would allow distributions to perform the long
> iteration chain once and then update rustc only from time to time.
> 
> Thanks
> Kind regards
>   Petr
> -- 
> Petr Cerny
> Mozilla/OpenSSH maintainer for SUSE Linux

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[banderson]

On Sunday, March 20, 2016 at 3:05:25 AM UTC-7, Henri Sivonen wrote:
> On Fri, Mar 18, 2016 at 6:45 PM, Mike Hommey  wrote:
> > On Fri, Mar 18, 2016 at 05:23:21PM +0200, Henri Sivonen wrote:
> >> You say you don't see #5 happening. Do you see #4 happening? If not,
> >> what do you see happening?
> >
> > At this point, I'm wondering if the best outcome wouldn't be 6) Distros
> > die. I'm almost not joking.
> 
> Non-jokingly, though, what resolution do you see?
> 
> >> Looking at 
> >> https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#browser-security
> >> , it seems that Debian users get a more up-to-date Blink than Gecko.
> >> If that policy is any indication, if ESR didn't exist, Gecko would get
> >> the same deal as Blink... In other words, it looks like Debian
> >> penalizes Gecko relative to Blink because ESR exists. :-(
> >
> > Well, at some point Blink wasn't even in stable. I'm actually surprised
> > that it is now.
> 
> The key thing is that it is now and updating it in a way that's an
> exception to Debian's general policy is now itself stated policy. So
> Debian has explicitly granted our main competitor the sort of policy
> exception that would be the best suited to resolving the problem at
> hand arising from Debian policy.
> 
> How do we get for rustc (as a build dependency of Firefox) the sort of
> policy exception that our main competitor already got?
> 
> > But as a matter of fact, Debian's old stable is not
> > receiving Blink/Chromium updates (it's stuck on 37), while it receives
> > updates for Iceweasel (it has 38.7 as or writing, will receive 38.8, and
> > 45.2 after that)
> 
> It appears that 45 ESR compiles with the GCC version that's in
> oldstable. What would have happened if that wasn't the case? What's
> the plan if during the support cycle of the current stable the next
> ESR doesn't build with the GCC version that shipped in current stable?
> 
> The comments on https://bugzilla.mozilla.org/show_bug.cgi?id=1175546
> suggest that Mozilla deferred relying on GCC bug fixes until after 45
> ESR in order for Debian not to have to backport a compiler for
> oldstable. Is that the case? That is, is Debian's policy already
> hindering Mozilla's ability to avoid C++ compiler bugs that the
> compiler upstream has fixed?
> 
> >> On Fri, Mar 18, 2016 at 1:01 PM, Sylvestre Ledru  
> >> wrote:
> >> > One dirty solution would be to ship rustc+llvm sources into Firefox 
> >> > sources when targeting stable distros.
> >> > But this increases the load on the maintainers by an order of magnitude 
> >> > (maintainers will have to manage rust & llvm)
> >> > and might not work if llvm starts requiring a more recent of gcc to 
> >> > build.
> >>
> >> Surely llvm can be built with clang, so you could include not only the
> >> source of every rustc release since Debian's freeze but the source
> >> code of every llvm and clang release since Debian's freeze... Except
> >> then you'd need a policy exception for the anti-bundling policy. If
> >> you need *some* policy exception no matter what, surely going for the
> >> most sensible exception (letting both the Firefox package and the
> >> rustc package update every six weeks within the LTS cycle) would make
> >> the most sense.
> >
> > A 6GB source tarball, for a build taking 200 hours just for one
> > architecture (I'm making up numbers, but I'm probably in the right
> > ballpark). Times 10 supported architectures. You'd better not have a
> > one-liner patch to apply on top of that.
> 
> Right. That would be an absurd outcome caused by Debian's policy. So
> how do we go about getting a policy exception to avoid absurdity?
> 
> > Note that this is why Blink/Chromium can get away with very frequent updates
> > in stable and not Iceweasel/Firefox:
> >
> > $ grep-dctrl -sPackage -FDepends chromium --and --not -FSource 
> > chromium-browser 
> > /var/lib/apt/lists/ftp.jp.debian.org_debian_dists_jessie_main_binary-amd64_Packages
> > | wc -l
> > 2
> >
> > (one is http://chromium-bsu.sourceforge.net/, the other is mozplugger,
> > which... sounds like a mistake... I think it's an NPAPI plugin)
> >
> >
> > $ grep-dctrl -sPackage -FDepends iceweasel --and --not -FSource iceweasel 
> > /var/lib/apt/lists/ftp.jp.debian.org_debian_dists_jessie_main_binary-amd64_Packages
> > | wc -l
> > 64
> 
> Are the dependent packages Firefox extensions? Now that Debian has
> already accepted upgrading to the next ESR during the lifecycle of
> stable, how are those dependent packages expected to survive that
> upgrade without being themselves updated?
> 
> On Fri, Mar 18, 2016 at 9:05 PM,   wrote:
> > One could similarly make distro packages of rustc specifically for building 
> > firefox (called 'rustc-fx' for example), and put them on the same ESR 
> > upgrade schedule as Firefox.
> 
> This is the approach Ubuntu takes with GCC in order rapid-release
> (non-ESR) Firefox for Ubuntu LTS.
> 
> 

Re: Linux distro readiness for Rust in Gecko

[banderson]

On Saturday, March 19, 2016 at 9:18:21 PM UTC-7, Cameron Kaiser wrote:
> On 3/19/16 5:27 AM, cosinusoida...@gmail.com wrote:
> > On Thursday, 17 March 2016 12:23:32 UTC, Henri Sivonen  wrote:
> >> On Thu, Mar 17, 2016 at 1:58 PM, Martin Stransky  
> >> wrote:
> >>> Is it possible to build Rust from sources before Firefox build executes?
> >>
> >> rustc is written in Rust, so if you only have source code and the
> >> compilers that are currently available in Fedora, you can't build
> >> rustc. At minimum, you need a "stage0" compiler, which is an archived
> >> older rustc binary. (rustc originally bootstrapped with OCaml, but
> >> building the whole history of Rust from the OCaml days to present
> >> every time Fedora builds Firefox seems excessively impractical.)
> >
> > Out of interest, would that actually involve building every single Linux 
> > snapshot from 
> > https://github.com/rust-lang/rust/blob/master/src/snapshots.txt in 
> > sequence? All 319 of them?
> 
> And are those the steps you'd actually have to take to bring Rust up 
> from scratch on a new platform?
> 
> Cameron Kaiser
> tier-3s in rain dept.

No, to bootstrap Rust you don't have to rebuild it's entire snapshot history. 
You cross compile from a platform that already has a compiler. The exact 
process is slightly different for every platform.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

Chris Peterson wrote:
> On 3/20/16 3:04 AM, Henri Sivonen wrote:
>> On Sat, Mar 19, 2016 at 2:27 PM,   wrote:
>>> > On Thursday, 17 March 2016 12:23:32 UTC, Henri Sivonen  wrote:
 >> (rustc originally bootstrapped with OCaml, but
 >> building the whole history of Rust from the OCaml days to present
 >> every time Fedora builds Firefox seems excessively impractical.)
>>> >
>>> > Out of interest, would that actually involve building every single Linux 
>>> > snapshot from 
>>> > https://github.com/rust-lang/rust/blob/master/src/snapshots.txt in 
>>> > sequence? All 319 of them?
>> Presumably, yes, if you want to bootstrap from OCaml. Maybe you could
>> skip some snapshots, but you don't know which ones in advance.
>> Fortunately, e.g. the Fedora policy doesn't require bootstrapping all
>> the way from OCaml.
> 
> How does Debian bootstrap other compilers like Go that are partially 
> self-hosted? Would a Rust compiler written in C/C++ (e.g. generated by 
> an LLVM back-end that emitted C/C++ code?) avoid the bootstrap policy 
> requirements?

That would definitely help (and not just for Debian), since
bootstrapping as far as possible is a generally preferred direction on
Linux, with the only exception of the basic toolchain (binutils, GCC),
libc and couple more packages that form a basic system (shell, make).
That is: you take a minimal working system, recompile new versions of
the packages with those from older release, then use the new minimal
system to build anything else.

If you can get sources that will allow building current rustc within say
3 build cycles, i.e.:

[C sources]---(C compiler)-->[rustc-intermediate]

[Rust sources]---(rustc-intermediate)-->[rustc-current]

[Rust sources]---(rustc-current)-->[rustc-current]

it will be just fine. Obviously 2 cycles would be nicer, but this even
with one added layer in between is manageable. If the distributed source
package contained both the rust sources and a bootstrapping compiler and
maybe even allowed building the final binary with a single command (a la
`[configure script]; make`) it would behave just like any other package.

Actually, this would be pretty much what GCC does: it bootstraps itself
in three phases, with just one `make` command. And although the scale is
much longer, compiling for example GCC-5.3.0 with GCC

The problem is having to go through some 300 iterations (even if one
would probably do that only once on each distribution release). Which as
far as I understand - and please do correct me if I'm wrong - would be
the case (presumably because rustc's code is progressively using its own
new features).

>From my point of view there are two options that would greatly help:

1) a bootstrapping rust compiler written in C  that would work in a
small number of rounds as has been described above. It could as well be
C++, Python, Perl or whatever can be commonly found in distributions -
the lower the better to keep the dependency tree smaller.

2) a more stable rustc codebase - stable in the sense that a recompile
would only be necessary on the order of magnitude of ESR releases (note:
I'm writing this without any knowledge about current rustc compile
requirements). This would allow distributions to perform the long
iteration chain once and then update rustc only from time to time.

Thanks
Kind regards
Petr
-- 
Petr Cerny
Mozilla/OpenSSH maintainer for SUSE Linux
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Sun, Mar 20, 2016 at 9:53 PM, Petr Cerny  wrote:
> I can't speak for Debian, but on SLE the solution we went for last time was
> packaging new-enough GCC for older supported code streams, very much like
> other packages (e.g. GTK) we had already needed way before.

This seems like a reasonable solution. As noted, it appears that
Ubuntu backports GCC, too.

>> The comments on https://bugzilla.mozilla.org/show_bug.cgi?id=1175546
>> suggest that Mozilla deferred relying on GCC bug fixes until after 45
>> ESR in order for Debian not to have to backport a compiler for
>> oldstable. Is that the case? That is, is Debian's policy already
>> hindering Mozilla's ability to avoid C++ compiler bugs that the
>> compiler upstream has fixed?
>
>
> TL;DR version: change whatever you need, but please do announce the intent
> loudly, clearly and early enough.

Great!

> Longer version:
>
> It definitely is not just Debian. With all due respect, Mozilla doesn't rule
> an isolated island - like all others it shares a jungle with other animals
> of varying sizes, shapes and habits.

Yes, but the Linux jungle depends on the outside for the browser
engines (Gecko, Blink and WebKit) that it relies on and the numbers
that Web developers look to assess the relevance of these engines
(which translates into whether sites are made in a way that these
engines work for Linux users) happens outside the Linux jungle. It's
not reasonable for a part of the already small jungle to be entitled
to limit the Gecko development inside and outside the jungle to old
compilers.

I want us to have good Linux support, I want obtaining our product to
be convenient for Linux users and I don't want conflicts with the
distros. I started this thread in order to avoid a situation where
nobody saw it coming that we'd be about to land a patch that broke the
distros' workflow by surprise. But consider: Rust as a technology
works the best on Linux. (IIRC, debugging info works better on Linux
than on Mac and it's pretty clear that Linux and Mac are better
matches for Rust development than Windows.) It's very messed up that
the tech that works the best on Linux is the hardest to ship to users
on Linux.

> I'm quite sure there are problems with
> other compilers/systems than just GCC/Linux, but due to their larger target
> user deployment and distribution model, they are being worked around. I know
> it is different (in *many* aspects), but just imagine you had to solve a
> similar issue with Microsoft or Apple (i.e. them distributing Firefox and
> strongly preferring to build it with what they use for building the rest of
> their systems with).

I don't think a comparison with Microsoft and Apple is in the distros'
favor. Microsoft's undocumented ABI has the practical effect of
locking us to their C++ compiler, but they don't make us use just the
C++ features and compiler bugs that were available when Windows XP
shipped. Apple promotes clang and we are at liberty to use clang from
llvm.org instead of Apple-shipped clang and we have the option to
patch clang if we need to.

As for imagining if Microsoft and Apple imposed requirements that they
don't: The important point is that they don't!

As for iOS, the requirement to use an Apple-designated language or
compiler is gone. The policy problem with Apple is that they ban the
class of product that we'd like to build on iOS. But AFAICT, using a
rustc version of the developer's choice to build other kinds of
products is already permitted on iOS.

> Yet back to the point: you certainly can change whichever tool you need or
> want, but the timing is crucial. Stable distribution maintainers are used to
> these things happening every once in a while, so if you announce it loudly a
> couple of releases ahead, it will be perfectly fine and nobody is going to
> complain

Not even Debian?

> Generally, I think it shouldn't really be perceived as: "Linux distributions
> are hindering our development", rather as being polite and respectful in a:
> "Let's not make much more work for them unless it is really, really
> necessary" manner. Which is exactly what shifting such a change from before
> to after a major ESR release is.

Considering how quickly Rust is progressing, chances are that the
Firefox will pick up new rustc releases faster than on a 10-month
cycle, so it will probably be safe to assume that every time there a
new major release becomes ESR, it'll require a newer rustc than the
previous ESR. But a mindset of "Linux distros can just ship ESR and
deal with rustc at 10-month intervals" isn't a great solution when the
distros have a non-enterprise audience or when the distros ship
Chromium every six weeks; see below.

> I think both sides want to achieve the same in the end: having a stable and
> secure Firefox.

It's not clear that that's the case when it comes to what "stable"
actually means. If "stable" means "doesn't crash", sure. If "stable"
means "out-of-date", the goals may not be aligned. 

Re: Linux distro readiness for Rust in Gecko

[Chris Peterson]

On 3/20/16 3:04 AM, Henri Sivonen wrote:

On Sat, Mar 19, 2016 at 2:27 PM,   wrote:

> On Thursday, 17 March 2016 12:23:32 UTC, Henri Sivonen  wrote:

>> (rustc originally bootstrapped with OCaml, but
>> building the whole history of Rust from the OCaml days to present
>> every time Fedora builds Firefox seems excessively impractical.)

>
> Out of interest, would that actually involve building every single Linux 
snapshot from https://github.com/rust-lang/rust/blob/master/src/snapshots.txt in 
sequence? All 319 of them?

Presumably, yes, if you want to bootstrap from OCaml. Maybe you could
skip some snapshots, but you don't know which ones in advance.
Fortunately, e.g. the Fedora policy doesn't require bootstrapping all
the way from OCaml.


How does Debian bootstrap other compilers like Go that are partially 
self-hosted? Would a Rust compiler written in C/C++ (e.g. generated by 
an LLVM back-end that emitted C/C++ code?) avoid the bootstrap policy 
requirements?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[rgarbas]

On Thursday, March 17, 2016 at 3:11:40 PM UTC+1, Nicolas B. Pierron wrote:
> On 03/17/2016 10:59 AM, Henri Sivonen wrote:
> > My concern is that Linux distributions tend to have a policy that
> > binary packages have to be buildable from source packages using the
> > tools that are available in the package repository of the distro.
> > Considering the current state of rustc availability in distro
> > repositories, it seems to me that we are headed towards a problem.
> 
> I guess one of the thing we could do is use an alternative solution, such as 
> an external package manager which can work side-by-side with the host, such 
> as zero-install, Portage, or Nix.
> 
> I already use Nix, as a package manager to manage the dependencies I need 
> for building Firefox [1].  One of the advantage I see with this solution is 
> that such Nix expression can be part of the repository [2] and be a fallback 
> for all the platforms where we have no support.
> 
> [1] https://github.com/nbp/firefox-build-env
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1115107
> 
> -- 
> Nicolas B. Pierron


Hi,

I'd like to back Nicolas here and point out few of this things that is hard to 
see when being new to Nix and evaluating 

 - Cross-platoform, Nix currently reliably works on any Linux distribution. 
Many are using Nix already as a replacement for homebrew on OSX. I've used Nix 
to package applications on Windows (via cygwin) and Nix since few months again 
compiles on FreeBSD (but package support is poor).

 - Multiple versions, you can install as many different version of packages as 
you need transparently, without affecting each other. Also it does not require 
any containerization solution to do this. 

 - Atomic, when re-installing or updating my development environment there is 
not as single moment where my environment is in a broken state.

 - Build reproducible and transparent source-binary model, you know exactly 
which version are used and you can reproduce the exact setup on many machines. 
And because of that you can extend existing build recepies easily and play with 
your environment.

 - Complete dependency tree, while on other distributions you rely on 3rdparty 
to provide images

 - Build tool for Deb/Rpm/... packages, primarily Nix can be considered as a 
build tool. If needed it can also produce .deb/.rpm/... binaries.


I've been using Nix for last 5 years and (since I know enough about Nix) I 
consider every current package manager broken (due to features mentioned 
above). It is hard to explain all the features of Nix in a short email, but 
with with all of the features mentioned above it would make sense to at least 
give it a try.


lp Rok
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[info]

On Thursday, March 17, 2016 at 3:11:40 PM UTC+1, Nicolas B. Pierron wrote:
> On 03/17/2016 10:59 AM, Henri Sivonen wrote:
> > My concern is that Linux distributions tend to have a policy that
> > binary packages have to be buildable from source packages using the
> > tools that are available in the package repository of the distro.
> > Considering the current state of rustc availability in distro
> > repositories, it seems to me that we are headed towards a problem.
> 
> I guess one of the thing we could do is use an alternative solution, such as 
> an external package manager which can work side-by-side with the host, such 
> as zero-install, Portage, or Nix.
> 
> I already use Nix, as a package manager to manage the dependencies I need 
> for building Firefox [1].  One of the advantage I see with this solution is 
> that such Nix expression can be part of the repository [2] and be a fallback 
> for all the platforms where we have no support.
> 
> [1] https://github.com/nbp/firefox-build-env
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1115107
> 

Hi,

I'd like to back Nicolas here and point out few of this things that is hard to 
see when being new to Nix and evaluating 

 - Cross-platoform, Nix currently reliably works on any Linux distribution. 
Many are using Nix already as a replacement for homebrew on OSX. I've used Nix 
to package applications on Windows (via cygwin) and Nix since few months again 
compiles on FreeBSD (but package support is poor).

 - Multiple versions, you can install as many different version of packages as 
you need transparently, without affecting each other. Also it does not require 
any containerization solution to do this. 

 - Atomic, when re-installing or updating my development environment there is 
not as single moment where my environment is in a broken state.

 - Build reproducible and transparent source-binary model, you know exactly 
which version are used and you can reproduce the exact setup on many machines. 
And because of that you can extend existing build recepies easily and play with 
your environment.

 - Complete dependency tree, while on other distributions you rely on 3rdparty 
to provide images

 - Build tool for Deb/Rpm/... packages, primarily Nix can be considered as a 
build tool. If needed it can also produce .deb/.rpm/... binaries.


I've been using Nix for last 5 years and (since I know enough about Nix) I 
consider every current package manager broken (due to features mentioned 
above). It is hard to explain all the features of Nix in a short email, but 
with with all of the features mentioned above it would make sense to at least 
give it a try.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Petr Cerny]

Hello,

my first post, so let me introduce myself briefly: I've been with SUSE 
Linux as Mozilla (currently Firefox+NSPR/NSS) maintainer for SLE (SUSE 
Linux Enterprise Server/Desktop) for some five years.


Henri Sivonen wrote:

But as a matter of fact, Debian's old stable is not
receiving Blink/Chromium updates (it's stuck on 37), while it receives
updates for Iceweasel (it has 38.7 as or writing, will receive 38.8, and
45.2 after that)


It appears that 45 ESR compiles with the GCC version that's in
oldstable. What would have happened if that wasn't the case? What's
the plan if during the support cycle of the current stable the next
ESR doesn't build with the GCC version that shipped in current stable?


I can't speak for Debian, but on SLE the solution we went for last time 
was packaging new-enough GCC for older supported code streams, very much 
like other packages (e.g. GTK) we had already needed way before. It 
wasn't entirely painless - to give you some context: both SUSE and 
RedHat have a 10+ years support, while Debian has 3 and Ubuntu LTS 5 years.



The comments on https://bugzilla.mozilla.org/show_bug.cgi?id=1175546
suggest that Mozilla deferred relying on GCC bug fixes until after 45
ESR in order for Debian not to have to backport a compiler for
oldstable. Is that the case? That is, is Debian's policy already
hindering Mozilla's ability to avoid C++ compiler bugs that the
compiler upstream has fixed?


TL;DR version: change whatever you need, but please do announce the 
intent loudly, clearly and early enough.


Longer version:

It definitely is not just Debian. With all due respect, Mozilla doesn't 
rule an isolated island - like all others it shares a jungle with other 
animals of varying sizes, shapes and habits. I'm quite sure there are 
problems with other compilers/systems than just GCC/Linux, but due to 
their larger target user deployment and distribution model, they are 
being worked around. I know it is different (in *many* aspects), but 
just imagine you had to solve a similar issue with Microsoft or Apple 
(i.e. them distributing Firefox and strongly preferring to build it with 
what they use for building the rest of their systems with).


Yet back to the point: you certainly can change whichever tool you need 
or want, but the timing is crucial. Stable distribution maintainers are 
used to these things happening every once in a while, so if you announce 
it loudly a couple of releases ahead, it will be perfectly fine and 
nobody is going to complain (especially when you give clear reasons), 
since there will be enough time to prepare for the change.


If you decided to introduce such a change couple of weeks before an ESR 
release date, you wouldn't want to go anywhere near the maintainers of 
older long-term-supported systems.


Generally, I think it shouldn't really be perceived as: "Linux 
distributions are hindering our development", rather as being polite and 
respectful in a: "Let's not make much more work for them unless it is 
really, really necessary" manner. Which is exactly what shifting such a 
change from before to after a major ESR release is. If you really need 
to "break things" in the above sense, please do communicate it 
thoroughly and early enough, for as small as the Firefox-on-Linux 
userbase might be by percentage, that percentage is still calculated 
from a quite large base.


I think both sides want to achieve the same in the end: having a stable 
and secure Firefox. That we may be coming from opposite sides (doing 
quick development with bleeding edge technologies versus supporting 
systems for 3+ years) need not matter that much. The key is trying to 
understand the other party's context (to give an example: while I had 
been rather sceptical about Rustification, after reading up a bit on the 
topic and some conversations with people who have deeper insight, many 
of my reservations were gone).


Thanks
Kind regards
Petr
--
Petr Cerny
Mozilla/OpenSSH maintainer for SUSE Linux
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Brian Smith]

Henri Sivonen  wrote:

> An example of this *not* being the case: I expect to have to import
> https://github.com/gz/rust-cpuid into Gecko in order to cater to the
> Mozilla-side policy sadness of having to support Windows XP users
> whose computers don't have SSE2.


With my Rust programmer hat on:

I recommend that you don't do that. Instead, have your Rust code call
Gecko's C/C++ CPUID code. Or set some global variable with the "has SSE2"
flag in your C++ code before the Rust code gets invoked, and have the Rust
code test the SSE2 flag.

It's not worth using unstable features, especially `asm!` which is
fundamentally flawed and shouldn't be stabilized, just to do something this
simple.

With my Esteemed Mozilla Alumni hat on:

It's absolutely ridiculous that Mozilla lets Debian and Red Hat choose what
tools Mozilla uses to build its software. Mozilla needs to use the best
tools for the job, and assist (mildly) in helping Debian and Red Hat cope
with whatever difficulties they incur due to that. Not just Rust, but
GCC/Clang and NSS and everything else.

And, same with ESR; even reading this sentence cost you more effort than
ESR is worth.

Love,
Brian
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Fri, Mar 18, 2016 at 6:45 PM, Mike Hommey  wrote:
> On Fri, Mar 18, 2016 at 05:23:21PM +0200, Henri Sivonen wrote:
>> You say you don't see #5 happening. Do you see #4 happening? If not,
>> what do you see happening?
>
> At this point, I'm wondering if the best outcome wouldn't be 6) Distros
> die. I'm almost not joking.

Non-jokingly, though, what resolution do you see?

>> Looking at 
>> https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#browser-security
>> , it seems that Debian users get a more up-to-date Blink than Gecko.
>> If that policy is any indication, if ESR didn't exist, Gecko would get
>> the same deal as Blink... In other words, it looks like Debian
>> penalizes Gecko relative to Blink because ESR exists. :-(
>
> Well, at some point Blink wasn't even in stable. I'm actually surprised
> that it is now.

The key thing is that it is now and updating it in a way that's an
exception to Debian's general policy is now itself stated policy. So
Debian has explicitly granted our main competitor the sort of policy
exception that would be the best suited to resolving the problem at
hand arising from Debian policy.

How do we get for rustc (as a build dependency of Firefox) the sort of
policy exception that our main competitor already got?

> But as a matter of fact, Debian's old stable is not
> receiving Blink/Chromium updates (it's stuck on 37), while it receives
> updates for Iceweasel (it has 38.7 as or writing, will receive 38.8, and
> 45.2 after that)

It appears that 45 ESR compiles with the GCC version that's in
oldstable. What would have happened if that wasn't the case? What's
the plan if during the support cycle of the current stable the next
ESR doesn't build with the GCC version that shipped in current stable?

The comments on https://bugzilla.mozilla.org/show_bug.cgi?id=1175546
suggest that Mozilla deferred relying on GCC bug fixes until after 45
ESR in order for Debian not to have to backport a compiler for
oldstable. Is that the case? That is, is Debian's policy already
hindering Mozilla's ability to avoid C++ compiler bugs that the
compiler upstream has fixed?

>> On Fri, Mar 18, 2016 at 1:01 PM, Sylvestre Ledru  wrote:
>> > One dirty solution would be to ship rustc+llvm sources into Firefox 
>> > sources when targeting stable distros.
>> > But this increases the load on the maintainers by an order of magnitude 
>> > (maintainers will have to manage rust & llvm)
>> > and might not work if llvm starts requiring a more recent of gcc to build.
>>
>> Surely llvm can be built with clang, so you could include not only the
>> source of every rustc release since Debian's freeze but the source
>> code of every llvm and clang release since Debian's freeze... Except
>> then you'd need a policy exception for the anti-bundling policy. If
>> you need *some* policy exception no matter what, surely going for the
>> most sensible exception (letting both the Firefox package and the
>> rustc package update every six weeks within the LTS cycle) would make
>> the most sense.
>
> A 6GB source tarball, for a build taking 200 hours just for one
> architecture (I'm making up numbers, but I'm probably in the right
> ballpark). Times 10 supported architectures. You'd better not have a
> one-liner patch to apply on top of that.

Right. That would be an absurd outcome caused by Debian's policy. So
how do we go about getting a policy exception to avoid absurdity?

> Note that this is why Blink/Chromium can get away with very frequent updates
> in stable and not Iceweasel/Firefox:
>
> $ grep-dctrl -sPackage -FDepends chromium --and --not -FSource 
> chromium-browser 
> /var/lib/apt/lists/ftp.jp.debian.org_debian_dists_jessie_main_binary-amd64_Packages
> | wc -l
> 2
>
> (one is http://chromium-bsu.sourceforge.net/, the other is mozplugger,
> which... sounds like a mistake... I think it's an NPAPI plugin)
>
>
> $ grep-dctrl -sPackage -FDepends iceweasel --and --not -FSource iceweasel 
> /var/lib/apt/lists/ftp.jp.debian.org_debian_dists_jessie_main_binary-amd64_Packages
> | wc -l
> 64

Are the dependent packages Firefox extensions? Now that Debian has
already accepted upgrading to the next ESR during the lifecycle of
stable, how are those dependent packages expected to survive that
upgrade without being themselves updated?

On Fri, Mar 18, 2016 at 9:05 PM,   wrote:
> One could similarly make distro packages of rustc specifically for building 
> firefox (called 'rustc-fx' for example), and put them on the same ESR upgrade 
> schedule as Firefox.

This is the approach Ubuntu takes with GCC in order rapid-release
(non-ESR) Firefox for Ubuntu LTS.

> As another idea: Rust could maintain an LTS branch designed to be upgraded in 
> conjunction with Firefox, that both keeps the language on a stable revision 
> while also exposing backported features that Firefox needs in a controlled 
> way. So Firefox would commit to being on e.g. 

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Thu, Mar 17, 2016 at 1:58 PM, Martin Stransky  wrote:
> Is it possible to build Rust from sources before Firefox build executes?

rustc is written in Rust, so if you only have source code and the
compilers that are currently available in Fedora, you can't build
rustc. At minimum, you need a "stage0" compiler, which is an archived
older rustc binary. (rustc originally bootstrapped with OCaml, but
building the whole history of Rust from the OCaml days to present
every time Fedora builds Firefox seems excessively impractical.)

Fedora's policy seems to prohibit continuous shipping of stage0 as a
Mozilla-provided binary, so it seems to me that either Fedora needs to
maintain rustc bootstrapped according to policy (bootstrapping using
Mozilla-provided stage0 once) or a policy waiver for Firefox is
needed.

See also https://internals.rust-lang.org/t/perfecting-rust-packaging/2623

-- 
Henri Sivonen
hsivo...@hsivonen.fi
https://hsivonen.fi/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Cameron Kaiser]

On 3/19/16 5:27 AM, cosinusoida...@gmail.com wrote:

On Thursday, 17 March 2016 12:23:32 UTC, Henri Sivonen  wrote:

On Thu, Mar 17, 2016 at 1:58 PM, Martin Stransky  wrote:

Is it possible to build Rust from sources before Firefox build executes?


rustc is written in Rust, so if you only have source code and the
compilers that are currently available in Fedora, you can't build
rustc. At minimum, you need a "stage0" compiler, which is an archived
older rustc binary. (rustc originally bootstrapped with OCaml, but
building the whole history of Rust from the OCaml days to present
every time Fedora builds Firefox seems excessively impractical.)


Out of interest, would that actually involve building every single Linux 
snapshot from https://github.com/rust-lang/rust/blob/master/src/snapshots.txt 
in sequence? All 319 of them?


And are those the steps you'd actually have to take to bring Rust up 
from scratch on a new platform?


Cameron Kaiser
tier-3s in rain dept.



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Mike Hommey]

On Sat, Mar 19, 2016 at 01:45:49AM +0900, Mike Hommey wrote:
> On Fri, Mar 18, 2016 at 05:23:21PM +0200, Henri Sivonen wrote:
> > You say you don't see #5 happening. Do you see #4 happening? If not,
> > what do you see happening?
> 
> At this point, I'm wondering if the best outcome wouldn't be 6) Distros
> die. I'm almost not joking.
> 
> > > LTS distros do update Firefox because there is no way they can support
> > > security updates on older releases (I've done it with 3.5 long enough to
> > > know it's not tractable). But they do it once a year (at every ESR bump),
> > > not every 6 weeks.
> > 
> > This is not the case for Ubuntu LTS. Even Ubuntu 12.04 gets a new
> > Firefox release every six weeks, and there is a package gcc-mozilla
> > that backports a GCC newer than the original GCC in 12.04 as a build
> > dependency:
> > http://archive.ubuntu.com/ubuntu/pool/main/f/firefox/firefox_45.0+build2-0ubuntu0.12.04.1.dsc
> > 
> > So, clearly, at least in the case of Ubuntu, there is precedent for 1)
> > updating Firefox every six weeks in LTS and 2) the Firefox package
> > having a build dependency on a compiler that's newer than the
> > compilers that originally shipped with the LTS system release.
> > 
> > When I started this thread, I thought the s/IceWeasel/Firefox/ change
> > in Debian involved Debian starting to ship Firefox the way Ubuntu
> > does. For clarity: Is that not the case and Debian will only ship ESR
> > but an ESR that's within Mozilla's support period? I can see how
> > shipping ESR is the closest approximation of compliance with a policy
> > to ship outdated software, but how does ESR address Debian's package
> > dependency issues? If the next ESR requires a compiler that's not in
> > the current Debian stable, what then?
> > 
> > Looking at 
> > https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#browser-security
> > , it seems that Debian users get a more up-to-date Blink than Gecko.
> > If that policy is any indication, if ESR didn't exist, Gecko would get
> > the same deal as Blink... In other words, it looks like Debian
> > penalizes Gecko relative to Blink because ESR exists. :-(
> 
> Well, at some point Blink wasn't even in stable. I'm actually surprised
> that it is now. But as a matter of fact, Debian's old stable is not
> receiving Blink/Chromium updates (it's stuck on 37), while it receives
> updates for Iceweasel (it has 38.7 as or writing, will receive 38.8, and
> 45.2 after that)

Note that this is why Blink/Chromium can get away with very frequent updates
in stable and not Iceweasel/Firefox:

$ grep-dctrl -sPackage -FDepends chromium --and --not -FSource chromium-browser 
/var/lib/apt/lists/ftp.jp.debian.org_debian_dists_jessie_main_binary-amd64_Packages
| wc -l
2

(one is http://chromium-bsu.sourceforge.net/, the other is mozplugger,
which... sounds like a mistake... I think it's an NPAPI plugin)


$ grep-dctrl -sPackage -FDepends iceweasel --and --not -FSource iceweasel 
/var/lib/apt/lists/ftp.jp.debian.org_debian_dists_jessie_main_binary-amd64_Packages
| wc -l
64

Iceweasel/Firefox is part of an ecosystem.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Tom Tromey]

> "Henri" == Henri Sivonen  writes:

Henri> Ubuntu and Fedora don't have any version of rustc anywhere at
Henri> all.

For Fedora it's in COPR, but of course that's not official, just
something to play with.

You can follow the Fedora Rust packaging process here:

https://bugzilla.redhat.com/show_bug.cgi?id=915043

Tom
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[banderson]

On Friday, March 18, 2016 at 4:53:58 AM UTC-7, Sylvestre Ledru wrote:
> Le 17/03/2016 à 21:30, Henri Sivonen a écrit :
> > On Thu, Mar 17, 2016 at 2:28 PM, Sylvestre Ledru  wrote:
> >> In Debian & Ubuntu,
> > I didn't find a rustc package on http://packages.ubuntu.com/ . What's
> > the situation on Ubuntu?
> I thought it was already synced for xenial. I just filled the request:
> https://bugs.launchpad.net/bugs/1559001
> >
> >> we use the official binaries provided to be able to build rust.
> > My same logic, can Mozilla-built rustc be used to build the Firefox
> > package as far as policy matters, with exceptions and waivers taken
> > into account, go? Or is this what's keeping rustc not getting past
> > testing?
> Debian stable will use the version of rustc at the time of the Debian freeze 
> (January 2017)
> >
> >
> > On Thu, Mar 17, 2016 at 2:47 PM, Sylvestre Ledru  wrote:
> >> One way which would make the life of Linux distro way easier would be
> >> to maintain the Firefox rust code in a way it could compile using older 
> >> rust compiler.
> > In order to be competitive, we need all leverage we can get from our
> > Mozilla magic sauce (Rust). I think it's unacceptable to limit our
> > ability to leverage Rust in Gecko by forgoing the ability to co-evolve
> > Rust and Gecko at a rapid pace.
> I understand your point, it is just conflicting with the goals of Debian and 
> Ubuntu LTS.
> Mozilla wants to move fast, distros wants to provide stable products...
> >
> >> Now, with my Debian/Ubuntu hat, maintaining rust backports to be able to 
> >> build new versions of Firefox on stable/LTS releases
> >> is not going to be easy
> > Could Firefox in Debian stable have a build dependency on rustc from
> > Debian unstable?
> Nope, this doesn't work this way. Packages must be built in the env they are 
> going to be used.
> Otherwise, it would lead to side effects (different version of libraries used 
> for build time and runtimes).
> 
> One dirty solution would be to ship rustc+llvm sources into Firefox sources 
> when targeting stable distros.
> But this increases the load on the maintainers by an order of magnitude 
> (maintainers will have to manage rust & llvm)
> and might not work if llvm starts requiring a more recent of gcc to build.
> However, this is really the last option distros will consider (and I am sure 
> Glandium will choke when he will read this).

This seems like a workable solution, as horrible as it is. One could similarly 
make distro packages of rustc specifically for building firefox (called 
'rustc-fx' for example), and put them on the same ESR upgrade schedule as 
Firefox.

As another idea: Rust could maintain an LTS branch designed to be upgraded in 
conjunction with Firefox, that both keeps the language on a stable revision 
while also exposing backported features that Firefox needs in a controlled way. 
So Firefox would commit to being on e.g. 1.12 for 3 years, but could import 
'future' features. Then LTS distros would put rustc LTS point releases on the 
same upgrade schedule as Firefox ESR upgrades.

For Firefox this would require a lot of discipline. For Rust it would be a 
massive challenge coming up with a maintainable scheme. I'm actually thinking 
Rust could support multiple language revisions with a single compiler in a way 
that would allow old revisions to receive continual bugfixes and integrate 
'future' features; and that would be reliable enough that LTS distros could 
make major version bumps to rustc alongside Firefox while still deploying the 
language revision compatible with their LTS. Difficult and crazy.



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Thu, Mar 17, 2016 at 2:28 PM, Sylvestre Ledru  wrote:
> In Debian & Ubuntu,

I didn't find a rustc package on http://packages.ubuntu.com/ . What's
the situation on Ubuntu?

> we use the official binaries provided to be able to build rust.

My same logic, can Mozilla-built rustc be used to build the Firefox
package as far as policy matters, with exceptions and waivers taken
into account, go? Or is this what's keeping rustc not getting past
testing?

> However, in the future, I have been told that we would be able to build rust 
> using rust version-1.

When is rustc expected to reach that level of bootstrapping predictability?

On Thu, Mar 17, 2016 at 2:47 PM, Sylvestre Ledru  wrote:
> One way which would make the life of Linux distro way easier would be
> to maintain the Firefox rust code in a way it could compile using older rust 
> compiler.

In order to be competitive, we need all leverage we can get from our
Mozilla magic sauce (Rust). I think it's unacceptable to limit our
ability to leverage Rust in Gecko by forgoing the ability to co-evolve
Rust and Gecko at a rapid pace.

> For example, imagine that the next Debian stable or Ubuntu LTS ships using 
> Rust 2.0,
> we would make sure that the new rust code still compile with Rust 2.0 
> (limiting our capability to use new
> language trick).

I think we must not allow ourselves to wait for a Debian or Ubuntu LTS
cycle before Rust improvements can be used in Gecko. We need to
leverage our advantages, such as Rust, fully to be competitive, and it
would be crazy to work at LTS pace when we are free to ship to the
majority of our users (on Windows and Mac) using whatever compiler
works best. (In general, regardless of building Gecko, I think it
would be harmful to the evolution of Rust at this stage for a version
to get stuck out there for the duration of a Debian or Ubuntu LTS
cycle. I agree with Brian Smith's sentiment at
https://internals.rust-lang.org/t/perfecting-rust-packaging/2623/67 .)

> Just like we do with C++...

And we fail to make best competitive use of our time and tools with
C++, too, when the latest clang, GCC and MSVC support a C++ feature,
we put effort into writing code and find out it has to rot without
landing because of old GCC.

> Now, with my Debian/Ubuntu hat, maintaining rust backports to be able to 
> build new versions of Firefox on stable/LTS releases
> is not going to be easy

Could Firefox in Debian stable have a build dependency on rustc from
Debian unstable?

On Thu, Mar 17, 2016 at 3:01 PM, Martin Stransky  wrote:
> Well, what about other arches? ppc(64), s390, ai64...?

All architectures currently supported by Fedora, Debian and Ubuntu
(but not Gentoo!) already have LLVM back ends. rustc hasn't been
ported to all those architectures, but the level of effort required
should be within reason considering that the LLVM back ends are there
and rustc has already been ported to target multiple ones of the
LLVM-supported ISAs.

If distro policy bans ongoing cross-compliation, I guess the distro
would need to replicate the Rust project's compiler compilation
version lineage on each architecture after bootstrapping with
cross-compilation.

On Thu, Mar 17, 2016 at 4:11 PM, Nicolas B. Pierron
 wrote:
> I guess one of the thing we could do is use an alternative solution, such as
> an external package manager which can work side-by-side with the host, such
> as zero-install, Portage, or Nix.

I don't see how that would be any better from the distro policy
perspective than using Mozilla-shipped rustc.

-- 
Henri Sivonen
hsivo...@hsivonen.fi
https://hsivonen.fi/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Sylvestre Ledru]

Le 17/03/2016 à 14:01, Martin Stransky a écrit :
> On 03/17/2016 01:47 PM, Sylvestre Ledru wrote:
>
> [...]
>
>> One way which would make the life of Linux distro way easier would be
>> to maintain the Firefox rust code in a way it could compile using older rust 
>> compiler.
>>
>> For example, imagine that the next Debian stable or Ubuntu LTS ships using 
>> Rust 2.0,
>> we would make sure that the new rust code still compile with Rust 2.0 
>> (limiting our capability to use new
>> language trick). Just like we do with C++...
>>
>> Now, with my Debian/Ubuntu hat, maintaining rust backports to be able to 
>> build new versions of Firefox on stable/LTS releases
>> is not going to be easy: rust needs recent versions of LLVM, LLVM is a 
>> complex beast (with regular bump the minimal version of gcc, etc).
>> And here, I am just mentioning i386 & amd64... (Debian/ubuntu don't have any 
>> rust builds for arm CPU).
>
> Well, what about other arches? ppc(64), s390, ai64...? 
For now, nothing has been done on Debian for this (except opening a bug [1]).
Using upstream binaries makes our life very easy for i386/amd64.
To start doing the porting job to other archs, being able to build using the 
previous version of the compiler is a requirement.
Otherwise, the cross compilation would be needed for each upstream version.

Sylvestre
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809316
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Sylvestre Ledru]

Le 17/03/2016 à 11:59, Henri Sivonen a écrit :
> Do we have a plan being executed to make sure that when Gecko
> components written in Rust ship on the release channel in
> Mozilla-produced builds the also ship in the Firefox packages of
> various Linux distributions?
[...]
>
> Can we get Debian stable, Ubuntu and Fedora to ship a rustc package as
> a build dependency for Firefox every six weeks like the Firefox
> package itself in order for Firefox not to miss the 6-week train when
> Rust code ships in Mozilla-built Firefox and for Firefox to be able to
> co-evolve with rustc instead of getting stuck to an old rustc due to
> distro LTS cycles? (I'm assuming that it won't be a big deal to get
> Gentoo to pick up the most recent rustc when the use case shows up.)
One way which would make the life of Linux distro way easier would be
to maintain the Firefox rust code in a way it could compile using older rust 
compiler.

For example, imagine that the next Debian stable or Ubuntu LTS ships using Rust 
2.0,
we would make sure that the new rust code still compile with Rust 2.0 (limiting 
our capability to use new
language trick). Just like we do with C++...

Now, with my Debian/Ubuntu hat, maintaining rust backports to be able to build 
new versions of Firefox on stable/LTS releases
is not going to be easy: rust needs recent versions of LLVM, LLVM is a complex 
beast (with regular bump the minimal version of gcc, etc).
And here, I am just mentioning i386 & amd64... (Debian/ubuntu don't have any 
rust builds for arm CPU).

Sylvestre

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Mike Hommey]

On Thu, Mar 17, 2016 at 04:59:29PM -0700, bander...@mozilla.com wrote:
> Hi,
> 
> This is a matter that concerns the Rust team a lot. From the Rust side
> we want to make it possible for Debian and Fedora (as a proxy for all
> distros) to be able to bootstrap off of their own rustc's and keep up
> with the Rust release process.
> 
> We have a [good idea][1] about how to accomplish that part, though I
> haven't followed through on the [implementation][2]. In particular, as
> noted in this thread, you can't bootstrap off of either the current
> version of Rust or the previous. This is actually a problem that could
> be fixed in about one release cycle. There's little excuse not to do it
> yesterday, though some on the team have lingering worry about having
> to wait 6 weeks to complete unstable language changes.
> 
> We have less understanding of what it will take to get the major
> distros to a) actually deploy rust in a stable release, b) keep rust updated
> every 6 weeks.

I don't see b) happening.

> > Do we have a plan being executed to make sure that when Gecko
> components written in Rust ship on the release channel in
> Mozilla-produced builds the also ship in the Firefox packages of
> various Linux distributions?
> 
> On the Rust side we have a plan to unblock the distros ability to
> package Rust [1]. As far as getting Rust into distros, we continue to
> go through cycles of finding blockers on the Rust side, fixing them,
> and then discovering what else is blocking the distros Unfortunately,
> after the initial push in the above threads, this has been on the
> backburner. I intend to push on it again next quarter.
> 
> > When is rustc expected to reach that level of bootstrapping predictability?
> 
> It should be feasible to make this change soon, but we'll need to
> discuss at the next core team meeting. I'll get back to you on that.
> 
> > I think it's unacceptable to limit our ability to leverage Rust in
> Gecko by forgoing the ability to co-evolve Rust and Gecko at a rapid
> pace.
> 
> Even after hurdles of getting stable rustc into distros is solved I
> think this is one of the most difficult issues. In particular, Firefox
> needs to be buildable on the *stable* Rust compiler in order for
> distros to build it. Rust's nightly compiler contains unstable
> features that we don't want used generally; it's the stable compiler
> that we promote and want distros to package. If Firefox requires
> nightly unstable features, then distros will be forced to package
> nightly Rust. If those distros users learn to expect that the nightly
> compiler is available then that could severely compromise Rust's
> strategy for evolving the language.
> 
> Note that there's a big diference between Firefox needing to
> continually update to a bleeding edge Rust, and Firefox using
> *unstable* Rust features. It's fine to update to a new Rust compiler
> every few weeks, even Rust nightlies, as long as Firefox doesn't turn
> on unstable features (which must be done explicitly).
> 
> > I think we must not allow ourselves to wait for a Debian or Ubuntu LTS
> cycle before Rust improvements can be used in Gecko.
> 
> > I agree with Brian Smith's sentiment at
> https://internals.rust-lang.org/t/perfecting-rust-packaging/2623/67
> 
> I agree that Rust being stuck on the distro LTS cycle is impracticle
> for real software.
> 
> If Rust produces 'universal' debs and rpms as suggested by @briansmith
> that would be enough for distro *users* to build Firefox, but it's not
> sufficient for the distro's themselves to keep their Firefox's up to
> date. I think Rust should do this but it isn't clear that it solves a
> critical problem.
> 
> Firefox's release model also has this tension with distros, and even
> LTS distros *do* update Firefox, right? Is there any chance Rust can
> recieve updates like Firefox?

LTS distros do update Firefox because there is no way they can support
security updates on older releases (I've done it with 3.5 long enough to
know it's not tractable). But they do it once a year (at every ESR bump),
not every 6 weeks.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Sylvestre Ledru]

Le 17/03/2016 à 21:30, Henri Sivonen a écrit :
> On Thu, Mar 17, 2016 at 2:28 PM, Sylvestre Ledru  wrote:
>> In Debian & Ubuntu,
> I didn't find a rustc package on http://packages.ubuntu.com/ . What's
> the situation on Ubuntu?
I thought it was already synced for xenial. I just filled the request:
https://bugs.launchpad.net/bugs/1559001
>
>> we use the official binaries provided to be able to build rust.
> My same logic, can Mozilla-built rustc be used to build the Firefox
> package as far as policy matters, with exceptions and waivers taken
> into account, go? Or is this what's keeping rustc not getting past
> testing?
Debian stable will use the version of rustc at the time of the Debian freeze 
(January 2017)
>
>
> On Thu, Mar 17, 2016 at 2:47 PM, Sylvestre Ledru  wrote:
>> One way which would make the life of Linux distro way easier would be
>> to maintain the Firefox rust code in a way it could compile using older rust 
>> compiler.
> In order to be competitive, we need all leverage we can get from our
> Mozilla magic sauce (Rust). I think it's unacceptable to limit our
> ability to leverage Rust in Gecko by forgoing the ability to co-evolve
> Rust and Gecko at a rapid pace.
I understand your point, it is just conflicting with the goals of Debian and 
Ubuntu LTS.
Mozilla wants to move fast, distros wants to provide stable products...
>
>> Now, with my Debian/Ubuntu hat, maintaining rust backports to be able to 
>> build new versions of Firefox on stable/LTS releases
>> is not going to be easy
> Could Firefox in Debian stable have a build dependency on rustc from
> Debian unstable?
Nope, this doesn't work this way. Packages must be built in the env they are 
going to be used.
Otherwise, it would lead to side effects (different version of libraries used 
for build time and runtimes).

One dirty solution would be to ship rustc+llvm sources into Firefox sources 
when targeting stable distros.
But this increases the load on the maintainers by an order of magnitude 
(maintainers will have to manage rust & llvm)
and might not work if llvm starts requiring a more recent of gcc to build.
However, this is really the last option distros will consider (and I am sure 
Glandium will choke when he will read this).

Sylvestre


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Sylvestre Ledru]

Le 17/03/2016 à 13:22, Henri Sivonen a écrit :
> On Thu, Mar 17, 2016 at 1:58 PM, Martin Stransky  wrote:
>> Is it possible to build Rust from sources before Firefox build executes?
> rustc is written in Rust, so if you only have source code and the
> compilers that are currently available in Fedora, you can't build
> rustc. At minimum, you need a "stage0" compiler, which is an archived
> older rustc binary. (rustc originally bootstrapped with OCaml, but
> building the whole history of Rust from the OCaml days to present
> every time Fedora builds Firefox seems excessively impractical.)
>
> Fedora's policy seems to prohibit continuous shipping of stage0 as a
> Mozilla-provided binary, so it seems to me that either Fedora needs to
> maintain rustc bootstrapped according to policy (bootstrapping using
> Mozilla-provided stage0 once) or a policy waiver for Firefox is
> needed.
In Debian & Ubuntu, we use the official binaries provided to be able to build 
rust.
Otherwise, it would be way too hard for a small gain (except to bootstrap other 
archs)

However, in the future, I have been told that we would be able to build rust 
using rust version-1.
Addressing this the core issue you mentioned (you need a special permission 
from the distro
to get the initial package).

Sylvestre

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[cosinusoidally]

On Thursday, 17 March 2016 12:23:32 UTC, Henri Sivonen  wrote:
> On Thu, Mar 17, 2016 at 1:58 PM, Martin Stransky  wrote:
> > Is it possible to build Rust from sources before Firefox build executes?
> 
> rustc is written in Rust, so if you only have source code and the
> compilers that are currently available in Fedora, you can't build
> rustc. At minimum, you need a "stage0" compiler, which is an archived
> older rustc binary. (rustc originally bootstrapped with OCaml, but
> building the whole history of Rust from the OCaml days to present
> every time Fedora builds Firefox seems excessively impractical.)

Out of interest, would that actually involve building every single Linux 
snapshot from https://github.com/rust-lang/rust/blob/master/src/snapshots.txt 
in sequence? All 319 of them?

Thanks
Liam Wilson
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Bobby Holley]

On Fri, Mar 18, 2016 at 12:05 PM,  wrote:

> For Firefox this would require a lot of discipline. For Rust it would be a
> massive challenge coming up with a maintainable scheme.


Frankly, I think the probability that either the Gecko or Rust teams would
commit significant portions of their limited engineering resources to this
issue is basically nil. This is a policy problem.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Henri Sivonen]

On Fri, Mar 18, 2016 at 1:10 AM, Nicolas B. Pierron
 wrote:
> This is not better from the point of view of distributions policy.
>
> This is better because you remove a lot of unknown variables from the
> equation, and rely on a real package manager to package and distribute
> software with its dependencies.

This seems off-topic to me. The matter at hand is dealing with
distros' (particularly Debian's and Fedora's) policies in such a
manner that there is no disruption to Firefox reaching users via these
channels while Mozilla leverages Mozilla-developed tech in Firefox at
the pace that makes sense for Mac, Windows and Mozilla's own Linux
builds. Building a distro's Firefox binary package with a package
manager that's not the distro's native one is very far off policy
compliance.

On Fri, Mar 18, 2016 at 1:59 AM,   wrote:
> In particular, as
> noted in this thread, you can't bootstrap off of either the current
> version of Rust or the previous. This is actually a problem that could
> be fixed in about one release cycle.

Nice! I didn't realize it's that close to happening. I think it would
be good to make "rustc stable N compiles with rustc stable N or rustc
stable N-1" happen, then. It would probably make life easier not just
for distro packaging but for non-x86/x86_64 ports as well, since those
could self-host in a predictable way.

> Even after hurdles of getting stable rustc into distros is solved I
> think this is one of the most difficult issues. In particular, Firefox
> needs to be buildable on the *stable* Rust compiler in order for
> distros to build it.

I've been assuming that Firefox would use the Rust "stable" channel
compiler on the Firefox "release" channel, but I don't know if that
has ever been really *decided* anywhere. Has it been decided?

> Rust's nightly compiler contains unstable
> features that we don't want used generally; it's the stable compiler
> that we promote and want distros to package. If Firefox requires
> nightly unstable features, then distros will be forced to package
> nightly Rust. If those distros users learn to expect that the nightly
> compiler is available then that could severely compromise Rust's
> strategy for evolving the language.

I think the best way to make sure that Firefox "release" channel
builds with Rust "stable" channel rustc is to make sure that the
theoretical notion of stability as the Rust project applies it and
practical de facto stability do indeed go pretty closely together.

An example of this *not* being the case: I expect to have to import
https://github.com/gz/rust-cpuid into Gecko in order to cater to the
Mozilla-side policy sadness of having to support Windows XP users
whose computers don't have SSE2. That crate requires the asm! macro,
which, according to
https://internals.rust-lang.org/t/stabilization-path-for-asm/2610 ,
has been "barely updated" since Rust 0.6 and no one seems to have a
concrete plan to redesign the feature, either. So the feature is de
facto stable, but it's not theoreticall stable and continues to be
unavailable in the "stable" channel.

> If Rust produces 'universal' debs and rpms as suggested by @briansmith
> that would be enough for distro *users* to build Firefox, but it's not
> sufficient for the distro's themselves to keep their Firefox's up to
> date. I think Rust should do this but it isn't clear that it solves a
> critical problem.

Indeed. I meant I agreed with this part of what Brian Smith said:
"Debian Stable and Red Hat are notorious for maintaining old versions
of packages way longer than anybody wants to support. With the great
amount of improvement to rustc and Cargo, I think we're actually 1 or
2 years away from being able to expect any Rust library or application
author to support any version of Rust older than the latest stable
release. I personally don't want to be bothered by Linux distros
asking me for free help in backporting changes or adding compatibility
hacks to support old versions of rustc and Cargo that they ship. I
imagine other people will feel similar."

> Firefox's release model also has this tension with distros, and even
> LTS distros *do* update Firefox, right?

Ubuntu LTS updates Firefox at the Mozilla every-six-weeks schedule.
>From Web sources, Fedora seems to, too, but I haven't actually
verified this empirically.

> Is there any chance Rust can
> recieve updates like Firefox?

>From my perspective, the best-case outcome of this thread would be exactly 
>that.

On Fri, Mar 18, 2016 at 3:08 AM, Mike Hommey  wrote:
>> We have less understanding of what it will take to get the major
>> distros to a) actually deploy rust in a stable release, b) keep rust updated
>> every 6 weeks.
>
> I don't see b) happening.

Why not if 1) Firefox has to update every six weeks to get security
updates and 2) rustc will become a build dependency for Firefox? That
is, why wouldn't whatever policy exception allows the Firefox 

Re: Linux distro readiness for Rust in Gecko

[Nicolas B. Pierron]

On 03/17/2016 10:59 AM, Henri Sivonen wrote:

My concern is that Linux distributions tend to have a policy that
binary packages have to be buildable from source packages using the
tools that are available in the package repository of the distro.
Considering the current state of rustc availability in distro
repositories, it seems to me that we are headed towards a problem.


I guess one of the thing we could do is use an alternative solution, such as 
an external package manager which can work side-by-side with the host, such 
as zero-install, Portage, or Nix.


I already use Nix, as a package manager to manage the dependencies I need 
for building Firefox [1].  One of the advantage I see with this solution is 
that such Nix expression can be part of the repository [2] and be a fallback 
for all the platforms where we have no support.


[1] https://github.com/nbp/firefox-build-env
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1115107

--
Nicolas B. Pierron
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Jack Moffitt]

> I've been assuming that Firefox would use the Rust "stable" channel
> compiler on the Firefox "release" channel, but I don't know if that
> has ever been really *decided* anywhere. Has it been decided?

Your assumption is correct. For gecko integration related work we use
the stable version of the language. However, this is explicitly not a
policy of Servo, where we make use of all the new features we can get
our hands on. These are not in conflict today*, but may someday be.
We'll handle that issue if we get there.

jack.

* rust-url can actually compile either way for example. We often
handle this by supporting both through the use of Cargo features
(amounting to conditional compilation).
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[banderson]

Hi,

This is a matter that concerns the Rust team a lot. From the Rust side
we want to make it possible for Debian and Fedora (as a proxy for all
distros) to be able to bootstrap off of their own rustc's and keep up
with the Rust release process.

We have a [good idea][1] about how to accomplish that part, though I
haven't followed through on the [implementation][2]. In particular, as
noted in this thread, you can't bootstrap off of either the current
version of Rust or the previous. This is actually a problem that could
be fixed in about one release cycle. There's little excuse not to do it
yesterday, though some on the team have lingering worry about having
to wait 6 weeks to complete unstable language changes.

We have less understanding of what it will take to get the major
distros to a) actually deploy rust in a stable release, b) keep rust updated
every 6 weeks.

> Do we have a plan being executed to make sure that when Gecko
components written in Rust ship on the release channel in
Mozilla-produced builds the also ship in the Firefox packages of
various Linux distributions?

On the Rust side we have a plan to unblock the distros ability to
package Rust [1]. As far as getting Rust into distros, we continue to
go through cycles of finding blockers on the Rust side, fixing them,
and then discovering what else is blocking the distros Unfortunately,
after the initial push in the above threads, this has been on the
backburner. I intend to push on it again next quarter.

> When is rustc expected to reach that level of bootstrapping predictability?

It should be feasible to make this change soon, but we'll need to
discuss at the next core team meeting. I'll get back to you on that.

> I think it's unacceptable to limit our ability to leverage Rust in
Gecko by forgoing the ability to co-evolve Rust and Gecko at a rapid
pace.

Even after hurdles of getting stable rustc into distros is solved I
think this is one of the most difficult issues. In particular, Firefox
needs to be buildable on the *stable* Rust compiler in order for
distros to build it. Rust's nightly compiler contains unstable
features that we don't want used generally; it's the stable compiler
that we promote and want distros to package. If Firefox requires
nightly unstable features, then distros will be forced to package
nightly Rust. If those distros users learn to expect that the nightly
compiler is available then that could severely compromise Rust's
strategy for evolving the language.

Note that there's a big diference between Firefox needing to
continually update to a bleeding edge Rust, and Firefox using
*unstable* Rust features. It's fine to update to a new Rust compiler
every few weeks, even Rust nightlies, as long as Firefox doesn't turn
on unstable features (which must be done explicitly).

> I think we must not allow ourselves to wait for a Debian or Ubuntu LTS
cycle before Rust improvements can be used in Gecko.

> I agree with Brian Smith's sentiment at
https://internals.rust-lang.org/t/perfecting-rust-packaging/2623/67

I agree that Rust being stuck on the distro LTS cycle is impracticle
for real software.

If Rust produces 'universal' debs and rpms as suggested by @briansmith
that would be enough for distro *users* to build Firefox, but it's not
sufficient for the distro's themselves to keep their Firefox's up to
date. I think Rust should do this but it isn't clear that it solves a
critical problem.

Firefox's release model also has this tension with distros, and even
LTS distros *do* update Firefox, right? Is there any chance Rust can
recieve updates like Firefox?

Again [this thread][1] contains the most up to date status.

[1]: https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767
[2]: 
https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/41
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Nicolas B. Pierron]

On 03/17/2016 08:30 PM, Henri Sivonen wrote:

On Thu, Mar 17, 2016 at 4:11 PM, Nicolas B. Pierron
 wrote:

I guess one of the thing we could do is use an alternative solution, such as
an external package manager which can work side-by-side with the host, such
as zero-install, Portage, or Nix.


I don't see how that would be any better from the distro policy
perspective than using Mozilla-shipped rustc.



This is not better from the point of view of distributions policy.

This is better because you remove a lot of unknown variables from the 
equation, and rely on a real package manager to package and distribute 
software with its dependencies.


If you think that doing it manually is better, I will ask you the following 
questions:

 - What libraries of the system do you depend on, which version?
 - What versions of python, cargo, …?
 - Is sh implemented as dash or bash ?
 - Do you need a custom version for Linux(es), Mac, BSD?
 - …
 - Do you have deterministic builds of rustc?

A portable package manager should be able to hide such variability for you, 
and reduce the burden of packaging.


Also note that these package managers can be installed in user-land, which 
is convenient if you have no root access on your computer.


--
Nicolas B. Pierron
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Martin Stransky]

On 03/17/2016 11:59 AM, Henri Sivonen wrote:

Do we have a plan being executed to make sure that when Gecko
components written in Rust ship on the release channel in
Mozilla-produced builds the also ship in the Firefox packages of
various Linux distributions?

If not, we really need to put such a plan in place, since it would be
bad to have a conflict between our ability to reach Linux users and
our ability to benefit from Rust quickly.

My concern is that Linux distributions tend to have a policy that
binary packages have to be buildable from source packages using the
tools that are available in the package repository of the distro.
Considering the current state of rustc availability in distro
repositories, it seems to me that we are headed towards a problem.

As far as I can tell, of the well-known distributions only Arch and
openSUSE have the current rustc in their package repositories (and the
openSUSE packages are categorized as "unstable"). Debian has it in
unstable and testing only. Gentoo is a couple of rustc releases out of
date. Ubuntu and Fedora don't have any version of rustc anywhere at
all.

At least in the case of Fedora, just shipping a rustc binary alongside
the Firefox source is explicitly not good enough for their policy:
https://fedoraproject.org/wiki/Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries


Yes, that's not possible in Fedora - we have similar problem with 
pre-built signed extensions from AMO.


Is it possible to build Rust from sources before Firefox build executes? 
That may be a solution for now and it ensures that the Rust is always up 
to date.


ma.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[Mike Hommey]

On Fri, Mar 18, 2016 at 05:23:21PM +0200, Henri Sivonen wrote:
> You say you don't see #5 happening. Do you see #4 happening? If not,
> what do you see happening?

At this point, I'm wondering if the best outcome wouldn't be 6) Distros
die. I'm almost not joking.

> > LTS distros do update Firefox because there is no way they can support
> > security updates on older releases (I've done it with 3.5 long enough to
> > know it's not tractable). But they do it once a year (at every ESR bump),
> > not every 6 weeks.
> 
> This is not the case for Ubuntu LTS. Even Ubuntu 12.04 gets a new
> Firefox release every six weeks, and there is a package gcc-mozilla
> that backports a GCC newer than the original GCC in 12.04 as a build
> dependency:
> http://archive.ubuntu.com/ubuntu/pool/main/f/firefox/firefox_45.0+build2-0ubuntu0.12.04.1.dsc
> 
> So, clearly, at least in the case of Ubuntu, there is precedent for 1)
> updating Firefox every six weeks in LTS and 2) the Firefox package
> having a build dependency on a compiler that's newer than the
> compilers that originally shipped with the LTS system release.
> 
> When I started this thread, I thought the s/IceWeasel/Firefox/ change
> in Debian involved Debian starting to ship Firefox the way Ubuntu
> does. For clarity: Is that not the case and Debian will only ship ESR
> but an ESR that's within Mozilla's support period? I can see how
> shipping ESR is the closest approximation of compliance with a policy
> to ship outdated software, but how does ESR address Debian's package
> dependency issues? If the next ESR requires a compiler that's not in
> the current Debian stable, what then?
> 
> Looking at 
> https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#browser-security
> , it seems that Debian users get a more up-to-date Blink than Gecko.
> If that policy is any indication, if ESR didn't exist, Gecko would get
> the same deal as Blink... In other words, it looks like Debian
> penalizes Gecko relative to Blink because ESR exists. :-(

Well, at some point Blink wasn't even in stable. I'm actually surprised
that it is now. But as a matter of fact, Debian's old stable is not
receiving Blink/Chromium updates (it's stuck on 37), while it receives
updates for Iceweasel (it has 38.7 as or writing, will receive 38.8, and
45.2 after that)

> On Fri, Mar 18, 2016 at 1:01 PM, Sylvestre Ledru  wrote:
> > Debian stable will use the version of rustc at the time of the Debian 
> > freeze (January 2017)
> 
> If Debian won't update rustc in Debian stable every six weeks, there's
> going to be the downside that I quoted from Brian Smith above. Same
> thing with Xenial if Ubuntu won't update the package every six weeks
> throughout the LTS cycle. :-(
> 
> I guess the upside is that then, in theory, the Debian Firefox source
> package could include the source code of every Rust "stable" channel
> rustc since Debian's freeze.
> 
> > One dirty solution would be to ship rustc+llvm sources into Firefox sources 
> > when targeting stable distros.
> > But this increases the load on the maintainers by an order of magnitude 
> > (maintainers will have to manage rust & llvm)
> > and might not work if llvm starts requiring a more recent of gcc to build.
> 
> Surely llvm can be built with clang, so you could include not only the
> source of every rustc release since Debian's freeze but the source
> code of every llvm and clang release since Debian's freeze... Except
> then you'd need a policy exception for the anti-bundling policy. If
> you need *some* policy exception no matter what, surely going for the
> most sensible exception (letting both the Firefox package and the
> rustc package update every six weeks within the LTS cycle) would make
> the most sense.

A 6GB source tarball, for a build taking 200 hours just for one
architecture (I'm making up numbers, but I'm probably in the right
ballpark). Times 10 supported architectures. You'd better not have a
one-liner patch to apply on top of that.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux distro readiness for Rust in Gecko

[aturon]

> On Fri, Mar 18, 2016 at 1:59 AM,   wrote:
> > In particular, as
> > noted in this thread, you can't bootstrap off of either the current
> > version of Rust or the previous. This is actually a problem that could
> > be fixed in about one release cycle.
> 
> Nice! I didn't realize it's that close to happening. I think it would
> be good to make "rustc stable N compiles with rustc stable N or rustc
> stable N-1" happen, then. It would probably make life easier not just
> for distro packaging but for non-x86/x86_64 ports as well, since those
> could self-host in a predictable way.

Agreed on all counts. We'll get this done ASAP.

On the broader distro policy questions: as Brian mentioned, we've been steadily 
making progress working with distros to get Rust shipped *at all*, as well as 
working through support for Rust applications. The need to update a package 
like Firefox with a newer version of rustc more rapidly than the distro itself 
hasn't been part of those discussions to date. This thread has helped clarify 
the constraints -- thanks for kicking it off. We'll make a renewed push working 
with our distro contacts with these goals in mind in Q2.

Cheers,
Aaron
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform