Re: nsIDownloadManager replaced by Downloads.jsm
Paolo Amadini wrote: A new about:config preference named browser.download.useJSTransfer enables the browser and the Downloads Panel to use the Downloads.jsm module instead of nsIDownloadManager as the back-end. The browser must be restarted for the preference to take effect. Support for this preference will be available in Nightly today or tomorrow. This means that it will be ready for testing in the Aurora channel starting from version 25, on August 8th. In the Firefox 26 release train, nsIDownloadManager will not be used anymore. The preference will be removed and there will be no way to revert to the old system that caused potential performance issues. We will finally be able to remove a lot of front-end code that is complex to maintain and only needed for backwards compatibility. I notice that this preference exists entirely in browser front-end code. So how are other Mozilla products expected to test Downloads.jsm? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Rethinking build defaults
Hi everyone, There's been a lot of traction recently about our builds getting slower and what we could do about it, and what not. Starting with bug 904979, I would like to change the way we're thinking about default flags and options. And I am therefore opening a discussion about it. The main thing bug 904979 does is to make release engineering builds (as well as linux distros, palemoon, icecat, you name it) use a special --enable-release configure flag to use flags that we deem necessary for a build of Firefox, the product. The flip side is that builds without this flag, which matches builds from every developer, obviously, would use flags that make the build faster. For the moment, on Linux systems, this means disabling identical code folding and dead code removal (which, while they make the binary size smaller, increase link time), and forcing the use of the gold linker when it's available but is not system default. With bug 905646, it will mean enabling -gsplit-dwarf when it's available, which make link time on linux really very much faster (4s on my machine instead of 30s). We could and should do the same kind of things for other platforms, with the goal of making linking libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster. If that works reliably, for instance, we should for instance use incremental linking. Please feel free to file Core::Build Config bugs for what you think would help on your favorite build platform (and if you do, for better tracking, make them depend on bug 904979). That being said, this is not the discussion I want to have here, that was merely an introduction. The web has grown in the past few years, and so has our code base, to support new technologies. As Nathan noted on his blog[1] disabling webrtc calls for great build time improvements. And I think it's something we should address by a switch in strategy. - How many people are working on webrtc code? - How many people are working on peripheral code that may affect webrtc? - How many people are building webrtc code they're not working on and not using? I'm fairly certain the answer to the above is that the latter population is much bigger than the other two, by probably more than an order of magnitude. So here's the discussion opener: why not make things like webrtc (I'm sure we can find many more[2]) opt-in instead of opt-out, for local, developer builds? What do you think are good candidates for such a switch? Mike 1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/ 2. and we can already start with ICU, because it's built and not even used. And to add injury to pain, it's currently built without parallelism (and the patch to make it not do so was backed out). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Rethinking build defaults
On Fri, Aug 16, 2013 at 05:43:22PM +0800, Andreas Gal wrote: First of all, thanks for raising this. Its definitely a problem that needs fixing. I am not convinced by your approach though. In a few months from now disabling WebRTC is like calling for the DOM or JS or CSS to be disabled in local developer builds. It will become a natural part of the core Web platform. I'm not running my local builds to browse the web. I bet a vast majority don't. I don't think what we build for local developer builds should be driven by what defines the web, but what brings useful results to developers in a timely manner. If disabling non central features can allow to build significantly faster, it's worth at least talking about it. No amount of widespread-ness on the web is going to make webrtc as central as js, layout, or gfx. If default is not an option, then maybe we can have --disable-all-the-things subsequently allowing opt-ins like --enable-webrtc. But I like to think that most people shouldn't have to edit a .mozconfig. I would like to propose the opposite approach: - Remove all conditional feature configuration from configure. WebRTC et al are always on. Features should be disabled dynamically (prefs), if at all. - Reduce configure settings to choice of OS and release or developer. With my Debian hat on, let me say this is both x86/x86-64/arm and Firefox/Firefox-OS centric. There are features that don't build on non mainstream architectures (hey webrtc, i'm looking at you), and while I do understand the horror it can be to some people that there could be a Firefox (^WIceweasel) build that doesn't support all the web, it's still better to have a browser that doesn't support everything than no browser at all (and considering i get bug reports from people using or trying to use iceweasel on e.g ppc or ia64, yes, there *are* people out there that like or would like to have a working browser, even if it doesn't make coffee). And Gecko is not used only to build web browsers (for how long?), so it makes sense for some features to be disableable at build time. - Require triple super-reviews (hand signed, in blood) for any changes to configure. - Make parts of the code base more modular and avoid super include files cross modules (hello LayoutUtils.h). Rationale: Its not slow for you to build WebRTC. Its slow for you to have it build over and over. Almost every time I pull from mozilla-central, someone touched configure and I have to rebuild from scratch, which is infuriating (argh!). Minimizing changes to configure and banning static defines for feature management would solve that. If we make sure major subsystems like WebRTC can stand on their own, you will take a hit building it one time, and then occasionally as the team lands changes. Its a pretty small team, so the amount of code they can possibly check in is actually pretty small. You will see churn all over the three when you pull, but you won't have to rebuild the entire Universe every time you pull. With my build-config peer hat on, let me say this may not be as much a problem as you feel it is. At least not one that requires drastic restrictive measures. There are three different things that make configure changes a pain: - Re-running configure even with no changes, changes some central files, and that means rebuilding a lot of things. I recently fixed bug 903341 (guess what, running configure meant rebuilding all webrtc), and bug 903321 (running configure meant rebuilding all js). There remains bug 903369, which makes us rebuild FFI, which makes us relink the js library and everything that statically links against it, including many tests and libxul. Another one that is yet to file is new and similar: running configure runs ICU's configure which makes us rebuild ICU, relinking the js library, and so on. When I get 903369 to work on windows, it will be easy to fix ICU in a similar way. - Adding or removing a AC_DEFINE in configure means modifying mozilla-config.h, which is included in every single file. Adding or removing an AC_DEFINE means rebuilding the entire tree. Now you know why I'm always reluctant when i see people adding an AC_DEFINE for something that is #ifdef'ed in 3 different directories. bug 902825 is on file about this. Note we're not adding or removing AC_DEFINEs /that/ often. - Adding or removing a AC_SUBST in configure means modifying autoconf.mk if the added variable has a value (if the variable doesn't have a value, it ends up in emptyvars.mk, which is treated differently). Autoconf.mk is essentially made a dependency of everything, so a change in autoconf.mk is going to rebuild everything. emptyvars.mk isn't. And now that I'm writing this, I'm thinking of an obvious solution to the adding/removing AC_SUBST problem: for features enabled by default, instead of landing AC_SUBST(ENABLE_FEATURE), land AC_SUBST(DISABLE_FEATURE). That would make the variable
Re: Rethinking build defaults
I think the key argument against this approach is that system components are never truly isolated. Sure, some of them can be compiled out and still produce a working system. That doesn't mean that testing without those components is going to have good test coverage. What I'm worried about, if we start disabling various modules, is that we're going to have regressions that go unnoticed on developer systems, blow up on m-i, and then take a _long_ time to track down. We already have m-i closed for about four hours a day as it is, frequently during prime working hours for a substantial fraction of Mozilla's contributors. Further varying developers' local build environments from those of the builders will only make this problem worse. /a On 8/16/13 04:32, Mike Hommey wrote: Hi everyone, There's been a lot of traction recently about our builds getting slower and what we could do about it, and what not. Starting with bug 904979, I would like to change the way we're thinking about default flags and options. And I am therefore opening a discussion about it. The main thing bug 904979 does is to make release engineering builds (as well as linux distros, palemoon, icecat, you name it) use a special --enable-release configure flag to use flags that we deem necessary for a build of Firefox, the product. The flip side is that builds without this flag, which matches builds from every developer, obviously, would use flags that make the build faster. For the moment, on Linux systems, this means disabling identical code folding and dead code removal (which, while they make the binary size smaller, increase link time), and forcing the use of the gold linker when it's available but is not system default. With bug 905646, it will mean enabling -gsplit-dwarf when it's available, which make link time on linux really very much faster (4s on my machine instead of 30s). We could and should do the same kind of things for other platforms, with the goal of making linking libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster. If that works reliably, for instance, we should for instance use incremental linking. Please feel free to file Core::Build Config bugs for what you think would help on your favorite build platform (and if you do, for better tracking, make them depend on bug 904979). That being said, this is not the discussion I want to have here, that was merely an introduction. The web has grown in the past few years, and so has our code base, to support new technologies. As Nathan noted on his blog[1] disabling webrtc calls for great build time improvements. And I think it's something we should address by a switch in strategy. - How many people are working on webrtc code? - How many people are working on peripheral code that may affect webrtc? - How many people are building webrtc code they're not working on and not using? I'm fairly certain the answer to the above is that the latter population is much bigger than the other two, by probably more than an order of magnitude. So here's the discussion opener: why not make things like webrtc (I'm sure we can find many more[2]) opt-in instead of opt-out, for local, developer builds? What do you think are good candidates for such a switch? Mike 1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/ 2. and we can already start with ICU, because it's built and not even used. And to add injury to pain, it's currently built without parallelism (and the patch to make it not do so was backed out). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform -- Adam Roach Principal Platform Engineer a...@mozilla.com +1 650 903 0800 x863 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Rethinking build defaults
- Original Message - I think the key argument against this approach is that system components are never truly isolated. Sure, some of them can be compiled out and still produce a working system. That doesn't mean that testing without those components is going to have good test coverage. What I'm worried about, if we start disabling various modules, is that we're going to have regressions that go unnoticed on developer systems, blow up on m-i, and then take a _long_ time to track down. We already have m-i closed for about four hours a day as it is, frequently during prime working hours for a substantial fraction of Mozilla's contributors. Further varying developers' local build environments from those of the builders will only make this problem worse. I imagine that most people only run tests locally that are related to the code they are working on, so it doesn't seem like it would make things any worse. Andrew /a On 8/16/13 04:32, Mike Hommey wrote: Hi everyone, There's been a lot of traction recently about our builds getting slower and what we could do about it, and what not. Starting with bug 904979, I would like to change the way we're thinking about default flags and options. And I am therefore opening a discussion about it. The main thing bug 904979 does is to make release engineering builds (as well as linux distros, palemoon, icecat, you name it) use a special --enable-release configure flag to use flags that we deem necessary for a build of Firefox, the product. The flip side is that builds without this flag, which matches builds from every developer, obviously, would use flags that make the build faster. For the moment, on Linux systems, this means disabling identical code folding and dead code removal (which, while they make the binary size smaller, increase link time), and forcing the use of the gold linker when it's available but is not system default. With bug 905646, it will mean enabling -gsplit-dwarf when it's available, which make link time on linux really very much faster (4s on my machine instead of 30s). We could and should do the same kind of things for other platforms, with the goal of making linking libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster. If that works reliably, for instance, we should for instance use incremental linking. Please feel free to file Core::Build Config bugs for what you think would help on your favorite build platform (and if you do, for better tracking, make them depend on bug 904979). That being said, this is not the discussion I want to have here, that was merely an introduction. The web has grown in the past few years, and so has our code base, to support new technologies. As Nathan noted on his blog[1] disabling webrtc calls for great build time improvements. And I think it's something we should address by a switch in strategy. - How many people are working on webrtc code? - How many people are working on peripheral code that may affect webrtc? - How many people are building webrtc code they're not working on and not using? I'm fairly certain the answer to the above is that the latter population is much bigger than the other two, by probably more than an order of magnitude. So here's the discussion opener: why not make things like webrtc (I'm sure we can find many more[2]) opt-in instead of opt-out, for local, developer builds? What do you think are good candidates for such a switch? Mike 1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/ 2. and we can already start with ICU, because it's built and not even used. And to add injury to pain, it's currently built without parallelism (and the patch to make it not do so was backed out). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform -- Adam Roach Principal Platform Engineer a...@mozilla.com +1 650 903 0800 x863 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Rethinking build defaults
On 16.08.2013 16:23, Andrew McCreight wrote: - Original Message - I think the key argument against this approach is that system components are never truly isolated. Sure, some of them can be compiled out and still produce a working system. That doesn't mean that testing without those components is going to have good test coverage. What I'm worried about, if we start disabling various modules, is that we're going to have regressions that go unnoticed on developer systems, blow up on m-i, and then take a _long_ time to track down. We already have m-i closed for about four hours a day as it is, frequently during prime working hours for a substantial fraction of Mozilla's contributors. Further varying developers' local build environments from those of the builders will only make this problem worse. I imagine that most people only run tests locally that are related to the code they are working on, so it doesn't seem like it would make things any worse. I would expect that the number of build failures increases if people disable parts of the code locally. Everything works for them, but when they push to m-i, things don't build any longer because the patches are incomplete. Best regards Thomas Andrew /a On 8/16/13 04:32, Mike Hommey wrote: Hi everyone, There's been a lot of traction recently about our builds getting slower and what we could do about it, and what not. Starting with bug 904979, I would like to change the way we're thinking about default flags and options. And I am therefore opening a discussion about it. The main thing bug 904979 does is to make release engineering builds (as well as linux distros, palemoon, icecat, you name it) use a special --enable-release configure flag to use flags that we deem necessary for a build of Firefox, the product. The flip side is that builds without this flag, which matches builds from every developer, obviously, would use flags that make the build faster. For the moment, on Linux systems, this means disabling identical code folding and dead code removal (which, while they make the binary size smaller, increase link time), and forcing the use of the gold linker when it's available but is not system default. With bug 905646, it will mean enabling -gsplit-dwarf when it's available, which make link time on linux really very much faster (4s on my machine instead of 30s). We could and should do the same kind of things for other platforms, with the goal of making linking libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster. If that works reliably, for instance, we should for instance use incremental linking. Please feel free to file Core::Build Config bugs for what you think would help on your favorite build platform (and if you do, for better tracking, make them depend on bug 904979). That being said, this is not the discussion I want to have here, that was merely an introduction. The web has grown in the past few years, and so has our code base, to support new technologies. As Nathan noted on his blog[1] disabling webrtc calls for great build time improvements. And I think it's something we should address by a switch in strategy. - How many people are working on webrtc code? - How many people are working on peripheral code that may affect webrtc? - How many people are building webrtc code they're not working on and not using? I'm fairly certain the answer to the above is that the latter population is much bigger than the other two, by probably more than an order of magnitude. So here's the discussion opener: why not make things like webrtc (I'm sure we can find many more[2]) opt-in instead of opt-out, for local, developer builds? What do you think are good candidates for such a switch? Mike 1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/ 2. and we can already start with ICU, because it's built and not even used. And to add injury to pain, it's currently built without parallelism (and the patch to make it not do so was backed out). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform -- Adam Roach Principal Platform Engineer a...@mozilla.com +1 650 903 0800 x863 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Rethinking build defaults
- Remove all conditional feature configuration from configure. WebRTC et al are always on. Features should be disabled dynamically (prefs), if at all. Note that Fabrice has seen sizable memory-usage wins on B2G by disabling non-web features such as printing and XUL. That alone is, in my mind, a very strong argument for keeping at least some of these configure options around. On Fri, Aug 16, 2013 at 2:43 AM, Andreas Gal andreas@gmail.com wrote: First of all, thanks for raising this. Its definitely a problem that needs fixing. I am not convinced by your approach though. In a few months from now disabling WebRTC is like calling for the DOM or JS or CSS to be disabled in local developer builds. It will become a natural part of the core Web platform. I would like to propose the opposite approach: - Remove all conditional feature configuration from configure. WebRTC et al are always on. Features should be disabled dynamically (prefs), if at all. - Reduce configure settings to choice of OS and release or developer. - Require triple super-reviews (hand signed, in blood) for any changes to configure. - Make parts of the code base more modular and avoid super include files cross modules (hello LayoutUtils.h). Rationale: Its not slow for you to build WebRTC. Its slow for you to have it build over and over. Almost every time I pull from mozilla-central, someone touched configure and I have to rebuild from scratch, which is infuriating (argh!). Minimizing changes to configure and banning static defines for feature management would solve that. If we make sure major subsystems like WebRTC can stand on their own, you will take a hit building it one time, and then occasionally as the team lands changes. Its a pretty small team, so the amount of code they can possibly check in is actually pretty small. You will see churn all over the three when you pull, but you won't have to rebuild the entire Universe every time you pull. What do you think? Andreas Mike Hommey wrote: Hi everyone, There's been a lot of traction recently about our builds getting slower and what we could do about it, and what not. Starting with bug 904979, I would like to change the way we're thinking about default flags and options. And I am therefore opening a discussion about it. The main thing bug 904979 does is to make release engineering builds (as well as linux distros, palemoon, icecat, you name it) use a special --enable-release configure flag to use flags that we deem necessary for a build of Firefox, the product. The flip side is that builds without this flag, which matches builds from every developer, obviously, would use flags that make the build faster. For the moment, on Linux systems, this means disabling identical code folding and dead code removal (which, while they make the binary size smaller, increase link time), and forcing the use of the gold linker when it's available but is not system default. With bug 905646, it will mean enabling -gsplit-dwarf when it's available, which make link time on linux really very much faster (4s on my machine instead of 30s). We could and should do the same kind of things for other platforms, with the goal of making linking libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster. If that works reliably, for instance, we should for instance use incremental linking. Please feel free to file Core::Build Config bugs for what you think would help on your favorite build platform (and if you do, for better tracking, make them depend on bug 904979). That being said, this is not the discussion I want to have here, that was merely an introduction. The web has grown in the past few years, and so has our code base, to support new technologies. As Nathan noted on his blog[1] disabling webrtc calls for great build time improvements. And I think it's something we should address by a switch in strategy. - How many people are working on webrtc code? - How many people are working on peripheral code that may affect webrtc? - How many people are building webrtc code they're not working on and not using? I'm fairly certain the answer to the above is that the latter population is much bigger than the other two, by probably more than an order of magnitude. So here's the discussion opener: why not make things like webrtc (I'm sure we can find many more[2]) opt-in instead of opt-out, for local, developer builds? What do you think are good candidates for such a switch? Mike 1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/ 2. and we can already start with ICU, because it's built and not even used. And to add injury to pain, it's currently built without parallelism (and the patch to make it not do so was backed out). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Rethinking build defaults
On 16/08/2013 15:14, Adam Roach wrote: What I'm worried about, if we start disabling various modules, is that we're going to have regressions that go unnoticed on developer systems, blow up on m-i, and then take a _long_ time to track down. They shouldn't take a long time to track down - a simple backout + let the dev build locally without the disable foo, bar baz by default pref set, will do the trick. It's also worth bearing in mind that we already have many cases of devs not testing locally (or not even building locally!) before pushing to inbound et al. There is a good chance that less people will fall into that group, if we speed up their local builds by default. As such, even if this proposal caused additional instances of broken builds when people pushed to inbound, it may still be outweighed by the reduced breakage from more devs testing locally. Best wishes, Ed ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
This week's award for net lines of code deleted
... goes to Blake Kaplan! He deleted 11,336 lines of code when he excised the old about:blank parser in bug 903912. btw, I didn't write any fancy scripts to scrape hg logs. github has pretty graphs for mozilla-central. However, the stats are skewed by merges and backouts. https://github.com/mozilla/mozilla-central/graphs/contributors?from=2013-08-09to=-12-31type=d chris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: nsIDownloadManager replaced by Downloads.jsm
On 16/08/2013 10.22, Neil wrote:
Paolo Amadini wrote:
A new about:config preference named browser.download.useJSTransfer
enables the browser and the Downloads Panel to use the Downloads.jsm
module instead of nsIDownloadManager as the back-end. The browser must
be restarted for the preference to take effect.
I notice that this preference exists entirely in browser front-end code.
So how are other Mozilla products expected to test Downloads.jsm?
The API itself can be used without changing the preference at all, if
you add items to the DownloadList using its methods. To switch downloads
started from content to use the new API, you should register its
nsITransfer component during profile-after-change:
const cid = Components.ID({1b4c85df-cbdd-4bb6-b04e-613caece083c});
Components.manager.QueryInterface(Ci.nsIComponentRegistrar)
.registerFactory(cid, , @mozilla.org/transfer;1,
null);
http://mxr.mozilla.org/mozilla-central/source/browser/components/downloads/src/DownloadsStartup.js#96
At some point, we'll add a build-time switch to enable this component.
Cheers,
Paolo
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: OS X: deprecate Apple clang 4.1?
On 8/15/13 2:50 PM, L. David Baron wrote: On Thursday 2013-08-15 14:11 -0700, Gregory Szorc wrote: I feel that having developers develop on the same toolchain as official builds (producing bit-identical builds if possible) would cut down on patch development costs due to reducing the frequency of failures resulting from discrepancies between the build environments. It also increases the costs of upgrading the toolchain and the costs of porting to new environments. Furthermore, it reduces the chance that we'll quickly catch real bugs in our code that only show up on some toolchains. Catching bugs quickly greatly reduces the cost of fixing them since the code is fresh in its author's mind. I'm going to push back against this a bit. Currently, the cost of supporting a new toolchain or build environment is highly distributed and not always planned since any developer at any time can use a varying toolchain or environment and experience breakage. Bugs are filed and we get distracted unbusting unsupported toolchains. There are benefits, sure, but there is a disruption cost here. By strictly limiting the number of supported environments and toolchains, we limit the overall effort required to support them. If I were to order our environments by their uniformity, I'd say Windows is the most uniform (we ship MozillaBuild and Microsoft ships a pre-built toolchain), followed by OS X, and finally Linux. While I don't have hard numbers, I'm reasonably confident saying environment variance is directly proportional to the number of bugs filed for build breakage. I'd say Windows build breakage not caught by our automation is rare. Linux and OS X build breakage, however, is comparatively common. I don't contend your point that supporting new environments or toolchains is a lot of work. However, I'm not sure if the cost is higher if we hold off, especially when you consider the advantages of performing the upgrade on our terms (as opposed to whenever people in the wild file bugs and cause mini fire drills by doing so). Anyway, we're not going to make any policy decisions about toolchain support at this time other than deprecating Apple Clang 4.1. I think it's best that we table this general discussion for another day. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
