Re: OS X: deprecate Apple clang 4.1?
On 2013-08-16 6:47 PM, Gregory Szorc wrote: On 8/15/13 2:50 PM, L. David Baron wrote: On Thursday 2013-08-15 14:11 -0700, Gregory Szorc wrote: I feel that having developers develop on the same toolchain as official builds (producing bit-identical builds if possible) would cut down on patch development costs due to reducing the frequency of failures resulting from discrepancies between the build environments. It also increases the costs of upgrading the toolchain and the costs of porting to new environments. Furthermore, it reduces the chance that we'll quickly catch real bugs in our code that only show up on some toolchains. Catching bugs quickly greatly reduces the cost of fixing them since the code is fresh in its author's mind. I'm going to push back against this a bit. Currently, the cost of supporting a new toolchain or build environment is highly distributed and not always planned since any developer at any time can use a varying toolchain or environment and experience breakage. Bugs are filed and we get distracted unbusting unsupported toolchains. There are benefits, sure, but there is a disruption cost here. I don't think the cost here is as high as you think, in most cases it is the developer who has encountered this problem who's fixing it in. By strictly limiting the number of supported environments and toolchains, we limit the overall effort required to support them. If I were to order our environments by their uniformity, I'd say Windows is the most uniform (we ship MozillaBuild and Microsoft ships a pre-built toolchain), followed by OS X, and finally Linux. While I don't have hard numbers, I'm reasonably confident saying environment variance is directly proportional to the number of bugs filed for build breakage. I'd say Windows build breakage not caught by our automation is rare. Linux and OS X build breakage, however, is comparatively common. I don't contend your point that supporting new environments or toolchains is a lot of work. However, I'm not sure if the cost is higher if we hold off, especially when you consider the advantages of performing the upgrade on our terms (as opposed to whenever people in the wild file bugs and cause mini fire drills by doing so). Hmm, I don't really know which issues we're trying to fix here. We explicitly break builds with toolchains that we know we don't want to support. We allow builds for all other toolchains. Expectiung that everybody uses the exact same toolchain as what we use in our infrastructure is not reasonable, and it hurts because of the reasons that David mentioned. We've always done this, and it just so happens that Apple clang 4.1 is a compiler that we're explicitly not interested in supporting. Has anybody else complained about anything here? :-) Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 19/08/13 11:30, Gijs Kruitbosch wrote: Note also that this is a big 'if' - on OS X, for instance, AFAIK we build with the 10.6 SDK on the official builds (because we still support 10.6). Getting a 10.6 SDK on a 10.8 mac is moderately painful (OTOH, there are real bugs that occur when building with the 10.6 SDK but not with the 10.8 SDK - and on yet another hand, there are probably bugs for which the inverse is true, too...) ~ Gijs Talking about Mac OS X, I'm currently stuck on SDK matters. Here is a link to the relevant comment: https://bugzilla.mozilla.org/show_bug.cgi?id=868648#c53 Quote: My past practice, using Xcode, was to use the latest SDK while targeting the earlier supported system for our products. And it worked as expected Objective C wise: when running on earlier systems, unimplemented methods just returned nil. Which we took care of checking at runtime. In our case this would mean using 10.8 SDK and targeting 10.6. I have no idea though how this translates in our own build system. And maybe we have a good reason not to do so. Markus' workaround is fine, but I'd still like to know why we're not using 10.8 SDK. André ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 15/08/13 23:11 , Gregory Szorc wrote: Let me rephrase your concern: if we could easily reproduce the official build environment on a developer's machine such that the discrepancy for build differences is minimized, why shouldn't we do that by default? Note also that this is a big 'if' - on OS X, for instance, AFAIK we build with the 10.6 SDK on the official builds (because we still support 10.6). Getting a 10.6 SDK on a 10.8 mac is moderately painful (OTOH, there are real bugs that occur when building with the 10.6 SDK but not with the 10.8 SDK - and on yet another hand, there are probably bugs for which the inverse is true, too...) ~ Gijs ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 8/15/13 2:50 PM, L. David Baron wrote: On Thursday 2013-08-15 14:11 -0700, Gregory Szorc wrote: I feel that having developers develop on the same toolchain as official builds (producing bit-identical builds if possible) would cut down on patch development costs due to reducing the frequency of failures resulting from discrepancies between the build environments. It also increases the costs of upgrading the toolchain and the costs of porting to new environments. Furthermore, it reduces the chance that we'll quickly catch real bugs in our code that only show up on some toolchains. Catching bugs quickly greatly reduces the cost of fixing them since the code is fresh in its author's mind. I'm going to push back against this a bit. Currently, the cost of supporting a new toolchain or build environment is highly distributed and not always planned since any developer at any time can use a varying toolchain or environment and experience breakage. Bugs are filed and we get distracted unbusting unsupported toolchains. There are benefits, sure, but there is a disruption cost here. By strictly limiting the number of supported environments and toolchains, we limit the overall effort required to support them. If I were to order our environments by their uniformity, I'd say Windows is the most uniform (we ship MozillaBuild and Microsoft ships a pre-built toolchain), followed by OS X, and finally Linux. While I don't have hard numbers, I'm reasonably confident saying environment variance is directly proportional to the number of bugs filed for build breakage. I'd say Windows build breakage not caught by our automation is rare. Linux and OS X build breakage, however, is comparatively common. I don't contend your point that supporting new environments or toolchains is a lot of work. However, I'm not sure if the cost is higher if we hold off, especially when you consider the advantages of performing the upgrade on our terms (as opposed to whenever people in the wild file bugs and cause mini fire drills by doing so). Anyway, we're not going to make any policy decisions about toolchain support at this time other than deprecating Apple Clang 4.1. I think it's best that we table this general discussion for another day. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On Thu, Aug 15, 2013 at 5:11 PM, Gregory Szorc wrote:
> On 8/15/13 12:18 PM, Ehsan Akhgari wrote:
>
>> On 2013-08-15 12:42 PM, Gregory Szorc wrote:
>>
>>> On 8/14/13 5:34 PM, Ehsan Akhgari wrote:
>>>
On Wed, Aug 14, 2013 at 4:44 PM, Joshua Cranmer 🐧
wrote:
On 8/14/2013 3:07 PM, Adam Roach wrote:
>
> Over the past few weeks, I've had the build completely break three
>> time
>> due to issues with Apple clang 4.1, which tells me that we're not
>> doing any
>> regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug 904108,
>> and the
>> fact that the current tip of m-i won't link with Apple clang 4.1).
>>
>> I'll note that the bugs I mention above are both working around actual
>> bugs in clang, not missing features.
>>
>> Any time I ask in #developers, the answer seems to be that our minimum
>> version for Apple clang is still 4.1. I would propose that (unless
>> we're
>> adapting some of our infra builders to check that we can at least
>> compile
>> and link with 4.1), we formally abandon 4.1 as a supported compiler.
>>
>>
> For what it's worth, the usual reference point for clang versions is
> synchronized with the LLVM versions, so the current tip-of-trunk is
> 3.3;
> when dealing with Clang compatibility, all version references refer
> to that
> internal number, so Clang 4.1 is a version which doesn't exist yet
> :-) .
> Guessing that "Apple clang 4.1" means "the clang shipped with Xcode
> 4.1",
> and using Wikipedia to get version info, this means that you're
> trying to
> use roughly Clang 3.0 ("based on LLVM 3.0svn"). We've never set a
> minimum
> Clang version to my knowledge for C++11 support (given that Clang has
> been
> ahead of the curve here), but I consider Clang < 3.1 unbuildable at
> least
> on Linux.
>
Should we make this explicit and fail builds with clang < 3.2?
>>>
>>> I'd go so far as to suggest that we explicitly require the Mozilla-built
>>> Clang package as defined by the tooltool manifest unless a
>>> --allow-unofficial-toolchain (or similar) configure flag is defined. I'm
>>> not sure if the package used by our builders is easily installed
>>> locally. So I think aggressively pruning the supported toolchain set is
>>> a good way to make development more pain free in the short term.
>>>
>>
>> Currently there is no (easy?) way for us to download those packages. Why
>> would we only allow Mozilla built packages in the first place though?
>> Compiling with newer clang is usually fine, it's old versions of clang
>> which cause these kinds of problems.
>>
>
> I feel that having developers develop on the same toolchain as official
> builds (producing bit-identical builds if possible) would cut down on patch
> development costs due to reducing the frequency of failures resulting from
> discrepancies between the build environments.
>
In addition to what David said, producing bit-identical builds in not
realistic on Windows and Linux where we do PGO. Also, the buildid
generation will probably make this impossible on a different level.
> If nothing else, having the toolchain remain in sync with something in the
> tree gives you the ability to go back and reproduce any old build. If we
> could easily reproduce builds over time, this would make a whole host of
> problems (such as bisecting, long-term analysis, decisions over data
> retention, etc) a lot easier.
>
For clang, we check in the LLVM SVN revision into the tree <
http://mxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/linux64/clang.manifest#3>,
so this is already possible (except for the fact that your build will not
be bit-identical, as mentioned above.)
> Let me rephrase your concern: if we could easily reproduce the official
> build environment on a developer's machine such that the discrepancy for
> build differences is minimized, why shouldn't we do that by default? About
> the only downside I can think of is lesser support for "other" build
> environments. But at the end of the day, Mozilla is only producing builds
> from N configurations, so I think it makes sense for us to keep the number
> of "cared about" configurations as close to N as possible.
>
In addition to what David said, if anything we should make it easier to
build with newer compilers. Also remember that with a
--allow-unofficial-toolchain, you'll make it pretty much impossible for
people to build master/aurora/beta/etc with the same local toolchain,
unless they edit all of their mozconfigs to have that, which defeats the
purpose.
Anyways, to get back to the topic of this thread, I filed bug 905876 to
disallow building with clang < 3.2.
--
Ehsan
Re: OS X: deprecate Apple clang 4.1?
On Thursday 2013-08-15 14:11 -0700, Gregory Szorc wrote: > I feel that having developers develop on the same toolchain as > official builds (producing bit-identical builds if possible) would > cut down on patch development costs due to reducing the frequency of > failures resulting from discrepancies between the build > environments. It also increases the costs of upgrading the toolchain and the costs of porting to new environments. Furthermore, it reduces the chance that we'll quickly catch real bugs in our code that only show up on some toolchains. Catching bugs quickly greatly reduces the cost of fixing them since the code is fresh in its author's mind. > If nothing else, having the toolchain remain in sync with something > in the tree gives you the ability to go back and reproduce any old > build. If we could easily reproduce builds over time, this would > make a whole host of problems (such as bisecting, long-term > analysis, decisions over data retention, etc) a lot easier. I think being able to reproduce things like nightlies is different from what we should force on developers. > Let me rephrase your concern: if we could easily reproduce the > official build environment on a developer's machine such that the > discrepancy for build differences is minimized, why shouldn't we do > that by default? I think it's great to enable that, but I think it's bad to require it, and I'd be somewhat concerned if everybody were using it. > About the only downside I can think of is lesser > support for "other" build environments. But at the end of the day, > Mozilla is only producing builds from N configurations, so I think > it makes sense for us to keep the number of "cared about" > configurations as close to N as possible. But, as I said above, those configurations evolve over time. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914) signature.asc Description: Digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 8/15/13 12:18 PM, Ehsan Akhgari wrote:
On 2013-08-15 12:42 PM, Gregory Szorc wrote:
On 8/14/13 5:34 PM, Ehsan Akhgari wrote:
On Wed, Aug 14, 2013 at 4:44 PM, Joshua Cranmer 🐧
wrote:
On 8/14/2013 3:07 PM, Adam Roach wrote:
Over the past few weeks, I've had the build completely break three
time
due to issues with Apple clang 4.1, which tells me that we're not
doing any
regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug 904108,
and the
fact that the current tip of m-i won't link with Apple clang 4.1).
I'll note that the bugs I mention above are both working around actual
bugs in clang, not missing features.
Any time I ask in #developers, the answer seems to be that our minimum
version for Apple clang is still 4.1. I would propose that (unless
we're
adapting some of our infra builders to check that we can at least
compile
and link with 4.1), we formally abandon 4.1 as a supported compiler.
For what it's worth, the usual reference point for clang versions is
synchronized with the LLVM versions, so the current tip-of-trunk is
3.3;
when dealing with Clang compatibility, all version references refer
to that
internal number, so Clang 4.1 is a version which doesn't exist yet
:-) .
Guessing that "Apple clang 4.1" means "the clang shipped with Xcode
4.1",
and using Wikipedia to get version info, this means that you're
trying to
use roughly Clang 3.0 ("based on LLVM 3.0svn"). We've never set a
minimum
Clang version to my knowledge for C++11 support (given that Clang has
been
ahead of the curve here), but I consider Clang < 3.1 unbuildable at
least
on Linux.
Should we make this explicit and fail builds with clang < 3.2?
I'd go so far as to suggest that we explicitly require the Mozilla-built
Clang package as defined by the tooltool manifest unless a
--allow-unofficial-toolchain (or similar) configure flag is defined. I'm
not sure if the package used by our builders is easily installed
locally. So I think aggressively pruning the supported toolchain set is
a good way to make development more pain free in the short term.
Currently there is no (easy?) way for us to download those packages. Why
would we only allow Mozilla built packages in the first place though?
Compiling with newer clang is usually fine, it's old versions of clang
which cause these kinds of problems.
I feel that having developers develop on the same toolchain as official
builds (producing bit-identical builds if possible) would cut down on
patch development costs due to reducing the frequency of failures
resulting from discrepancies between the build environments.
If nothing else, having the toolchain remain in sync with something in
the tree gives you the ability to go back and reproduce any old build.
If we could easily reproduce builds over time, this would make a whole
host of problems (such as bisecting, long-term analysis, decisions over
data retention, etc) a lot easier.
Let me rephrase your concern: if we could easily reproduce the official
build environment on a developer's machine such that the discrepancy for
build differences is minimized, why shouldn't we do that by default?
About the only downside I can think of is lesser support for "other"
build environments. But at the end of the day, Mozilla is only producing
builds from N configurations, so I think it makes sense for us to keep
the number of "cared about" configurations as close to N as possible.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 2013-08-15 12:42 PM, Gregory Szorc wrote:
On 8/14/13 5:34 PM, Ehsan Akhgari wrote:
On Wed, Aug 14, 2013 at 4:44 PM, Joshua Cranmer 🐧
wrote:
On 8/14/2013 3:07 PM, Adam Roach wrote:
Over the past few weeks, I've had the build completely break three time
due to issues with Apple clang 4.1, which tells me that we're not
doing any
regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug 904108,
and the
fact that the current tip of m-i won't link with Apple clang 4.1).
I'll note that the bugs I mention above are both working around actual
bugs in clang, not missing features.
Any time I ask in #developers, the answer seems to be that our minimum
version for Apple clang is still 4.1. I would propose that (unless
we're
adapting some of our infra builders to check that we can at least
compile
and link with 4.1), we formally abandon 4.1 as a supported compiler.
For what it's worth, the usual reference point for clang versions is
synchronized with the LLVM versions, so the current tip-of-trunk is 3.3;
when dealing with Clang compatibility, all version references refer
to that
internal number, so Clang 4.1 is a version which doesn't exist yet :-) .
Guessing that "Apple clang 4.1" means "the clang shipped with Xcode
4.1",
and using Wikipedia to get version info, this means that you're
trying to
use roughly Clang 3.0 ("based on LLVM 3.0svn"). We've never set a
minimum
Clang version to my knowledge for C++11 support (given that Clang has
been
ahead of the curve here), but I consider Clang < 3.1 unbuildable at
least
on Linux.
Should we make this explicit and fail builds with clang < 3.2?
I'd go so far as to suggest that we explicitly require the Mozilla-built
Clang package as defined by the tooltool manifest unless a
--allow-unofficial-toolchain (or similar) configure flag is defined. I'm
not sure if the package used by our builders is easily installed
locally. So I think aggressively pruning the supported toolchain set is
a good way to make development more pain free in the short term.
Currently there is no (easy?) way for us to download those packages.
Why would we only allow Mozilla built packages in the first place
though? Compiling with newer clang is usually fine, it's old versions
of clang which cause these kinds of problems.
Cheers,
Ehsan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 8/14/13 5:34 PM, Ehsan Akhgari wrote:
On Wed, Aug 14, 2013 at 4:44 PM, Joshua Cranmer 🐧 wrote:
On 8/14/2013 3:07 PM, Adam Roach wrote:
Over the past few weeks, I've had the build completely break three time
due to issues with Apple clang 4.1, which tells me that we're not doing any
regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug 904108, and the
fact that the current tip of m-i won't link with Apple clang 4.1).
I'll note that the bugs I mention above are both working around actual
bugs in clang, not missing features.
Any time I ask in #developers, the answer seems to be that our minimum
version for Apple clang is still 4.1. I would propose that (unless we're
adapting some of our infra builders to check that we can at least compile
and link with 4.1), we formally abandon 4.1 as a supported compiler.
For what it's worth, the usual reference point for clang versions is
synchronized with the LLVM versions, so the current tip-of-trunk is 3.3;
when dealing with Clang compatibility, all version references refer to that
internal number, so Clang 4.1 is a version which doesn't exist yet :-) .
Guessing that "Apple clang 4.1" means "the clang shipped with Xcode 4.1",
and using Wikipedia to get version info, this means that you're trying to
use roughly Clang 3.0 ("based on LLVM 3.0svn"). We've never set a minimum
Clang version to my knowledge for C++11 support (given that Clang has been
ahead of the curve here), but I consider Clang < 3.1 unbuildable at least
on Linux.
Should we make this explicit and fail builds with clang < 3.2?
I'd go so far as to suggest that we explicitly require the Mozilla-built
Clang package as defined by the tooltool manifest unless a
--allow-unofficial-toolchain (or similar) configure flag is defined. I'm
not sure if the package used by our builders is easily installed
locally. So I think aggressively pruning the supported toolchain set is
a good way to make development more pain free in the short term.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On Wed, Aug 14, 2013 at 4:44 PM, Joshua Cranmer 🐧 wrote:
> On 8/14/2013 3:07 PM, Adam Roach wrote:
>
>> Over the past few weeks, I've had the build completely break three time
>> due to issues with Apple clang 4.1, which tells me that we're not doing any
>> regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug 904108, and the
>> fact that the current tip of m-i won't link with Apple clang 4.1).
>>
>> I'll note that the bugs I mention above are both working around actual
>> bugs in clang, not missing features.
>>
>> Any time I ask in #developers, the answer seems to be that our minimum
>> version for Apple clang is still 4.1. I would propose that (unless we're
>> adapting some of our infra builders to check that we can at least compile
>> and link with 4.1), we formally abandon 4.1 as a supported compiler.
>>
>
> For what it's worth, the usual reference point for clang versions is
> synchronized with the LLVM versions, so the current tip-of-trunk is 3.3;
> when dealing with Clang compatibility, all version references refer to that
> internal number, so Clang 4.1 is a version which doesn't exist yet :-) .
> Guessing that "Apple clang 4.1" means "the clang shipped with Xcode 4.1",
> and using Wikipedia to get version info, this means that you're trying to
> use roughly Clang 3.0 ("based on LLVM 3.0svn"). We've never set a minimum
> Clang version to my knowledge for C++11 support (given that Clang has been
> ahead of the curve here), but I consider Clang < 3.1 unbuildable at least
> on Linux.
Should we make this explicit and fail builds with clang < 3.2?
--
Ehsan
Re: OS X: deprecate Apple clang 4.1?
On 8/14/2013 3:07 PM, Adam Roach wrote:
Over the past few weeks, I've had the build completely break three
time due to issues with Apple clang 4.1, which tells me that we're not
doing any regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug
904108, and the fact that the current tip of m-i won't link with Apple
clang 4.1).
I'll note that the bugs I mention above are both working around actual
bugs in clang, not missing features.
Any time I ask in #developers, the answer seems to be that our minimum
version for Apple clang is still 4.1. I would propose that (unless
we're adapting some of our infra builders to check that we can at
least compile and link with 4.1), we formally abandon 4.1 as a
supported compiler.
For what it's worth, the usual reference point for clang versions is
synchronized with the LLVM versions, so the current tip-of-trunk is 3.3;
when dealing with Clang compatibility, all version references refer to
that internal number, so Clang 4.1 is a version which doesn't exist yet
:-) . Guessing that "Apple clang 4.1" means "the clang shipped with
Xcode 4.1", and using Wikipedia to get version info, this means that
you're trying to use roughly Clang 3.0 ("based on LLVM 3.0svn"). We've
never set a minimum Clang version to my knowledge for C++11 support
(given that Clang has been ahead of the curve here), but I consider
Clang < 3.1 unbuildable at least on Linux.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
cc-ing dev-builds to see what the BuildConfig folks think. tc John. = On Wed Aug 14 13:07:14 2013, Adam Roach wrote: > Over the past few weeks, I've had the build completely break three > time due to issues with Apple clang 4.1, which tells me that we're not > doing any regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug > 904108, and the fact that the current tip of m-i won't link with Apple > clang 4.1). > > I'll note that the bugs I mention above are both working around actual > bugs in clang, not missing features. > > Any time I ask in #developers, the answer seems to be that our minimum > version for Apple clang is still 4.1. I would propose that (unless > we're adapting some of our infra builders to check that we can at > least compile and link with 4.1), we formally abandon 4.1 as a > supported compiler. > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
OS X: deprecate Apple clang 4.1?
Over the past few weeks, I've had the build completely break three time due to issues with Apple clang 4.1, which tells me that we're not doing any regular builds with Apple clang 4.1 (c.f. Bug 892594, Bug 904108, and the fact that the current tip of m-i won't link with Apple clang 4.1). I'll note that the bugs I mention above are both working around actual bugs in clang, not missing features. Any time I ask in #developers, the answer seems to be that our minimum version for Apple clang is still 4.1. I would propose that (unless we're adapting some of our infra builders to check that we can at least compile and link with 4.1), we formally abandon 4.1 as a supported compiler. -- Adam Roach Principal Platform Engineer a...@mozilla.com +1 650 903 0800 x863 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
