Re: Closure of trunk trees - owners for bugs needed
And to bring this full circle... >> - Finish fixing the social API leaks: >> https://bugzilla.mozilla.org/show_bug.cgi?id=933551. > > Fixed on mozilla-central. Not yet backported to other repos. I asked > ttaubert to nominate them for backporting. ttaubert pointed out that these just help with the shutdown leak detector, and so don't need backporting. >> - Improve the shutdown leak detection situation: >> https://bugzilla.mozilla.org/show_bug.cgi?id=932898. > > Two of three patches have r+, and the third is awaiting review by ted. > Not sure if these patches will need backporting or not. The patches in this bug have landed, and don't need backporting. All the relevant bugs have been resolved and there is nothing left to do w.r.t. this tree closure. And although the tree closure was annoying, we exited with (a) multiple leaks fixed, (b) some improved tests, and (c) improved shutdown leak detection. Not bad! Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
On Fri, Nov 1, 2013 at 12:52 PM, Nicholas Nethercote wrote: > > - Finish fixing the devtools leaks: > https://bugzilla.mozilla.org/show_bug.cgi?id=932880. Fixed on mozilla-central and backported to aurora, beta, and b2g26_v1_2. Nothing more to do. > - Finish fixing the social API leaks: > https://bugzilla.mozilla.org/show_bug.cgi?id=933551. Fixed on mozilla-central. Not yet backported to other repos. I asked ttaubert to nominate them for backporting. > - Improve the shutdown leak detection situation: > https://bugzilla.mozilla.org/show_bug.cgi?id=932898. Two of three patches have r+, and the third is awaiting review by ted. Not sure if these patches will need backporting or not. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
On Fri, Nov 1, 2013 at 7:26 AM, Andrew McCreight wrote: > > - Original Message - >> Is there anything else we can do to prevent this from happening again? > > It should be possible to track the peak number of live DOM windows in a test. > The current number of live windows is printed out in the log as the number > right after --DOMWINDOW == or ++DOMWINDOW == . > > From there, the question is what to do with that data. One possibility would > be to track it on tree-management. Somebody would have to keep an eye on it. > New tests that cause this number to go up by a lot could be backed out. That's a good idea, but before the fixes landed, for Mochi-2 the number at shutdown ranged from ~650 to ~1050. I suspect that much variation would cause problems for the tree-mgmt regression heuristics? Maybe after the fixes the variation is less? Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
On 11/01/2013 07:55 AM, Nicholas Nethercote wrote: I have (slightly optimistically) started writing a post-mortem of this closure, analyzing what went wrong and why, and how we might avoid it in the future: https://etherpad.mozilla.org/mEB0H50ZjX FWIW, I added the following TL;DR to the document, which reflects my understanding of the situation. Win7 M2 and Mbc tests were OOMing frequently at shutdown because too many DOM windows were open. This was due to a combination of: (a) multiple badly written tests, (b) multiple social API leaks, (c) multiple devtool leaks. Bug 932898 will improve our shutdown leak detection. Bug 932900 will (if implemented) prevent some of these leaks(?). Is there anything else we can do to prevent this from happening again? Nick We should add some checks that hiddenWindow doesn't contain anything unexpected when closing FF. The DOM tree and window scope should hopefully look the sama as when starting FF. -Olli ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
- Original Message - > Is there anything else we can do to prevent this from happening again? It should be possible to track the peak number of live DOM windows in a test. The current number of live windows is printed out in the log as the number right after --DOMWINDOW == or ++DOMWINDOW == . >From there, the question is what to do with that data. One possibility would >be to track it on tree-management. Somebody would have to keep an eye on it. >New tests that cause this number to go up by a lot could be backed out. I >feel like in the end, the real problem keeping the tree green was a handful of >individual tests that were creating hundreds of windows, so the addition of >tests like that should be fairly easy to do. The problem getting slowly worse >until suddenly it falls over is a concern, but maybe because we're tracking >the maximum it is less of an issue? Another way to expose this information would be to probabilistically make the test orange based on how close we are to some per-test peak window limit. The idea here as we slowly creep closer to the limit, the test would slowly become more orange. At a certain point, sheriffs would notice, and get somebody to look into what is happening, before the tree falls over entirely. You'd want to track this orange on a per-test-suite basis (I guess by including the name of the suite in the orange), and tuning the probability function would be tricky. Plus the sheriff's may not be a fan of graduated-sheriff-annoyance as a strategy, so maybe the more conventional tree-management mail would be better. Andrew ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
On 01/11/2013 01:58, Nicholas Nethercote wrote: One more: lots of patches will need to be backported to Aurora and Beta. I've set the appropriate tracking flags on the bugs that I think need it, but please double-check ones you know about in case I've missed any. Nick I'll keep an eye on these as bugs get fixed & make sure we backport to aurora/beta as needed :-) Best wishes, Ed ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
> I have (slightly optimistically) started writing a post-mortem of this > closure, analyzing what went wrong and why, and how we might avoid it > in the future: > > https://etherpad.mozilla.org/mEB0H50ZjX FWIW, I added the following TL;DR to the document, which reflects my understanding of the situation. > Win7 M2 and Mbc tests were OOMing frequently at shutdown because too many > DOM windows were open. This was due to a combination of: (a) multiple badly > written tests, (b) multiple social API leaks, (c) multiple devtool leaks. > Bug 932898 > will improve our shutdown leak detection. Bug 932900 will (if implemented) > prevent > some of these leaks(?). Is there anything else we can do to prevent this from happening again? Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
On Thu, Oct 31, 2013 at 6:52 PM, Nicholas Nethercote wrote: > > Remaining things I'm aware of: One more: lots of patches will need to be backported to Aurora and Beta. I've set the appropriate tracking flags on the bugs that I think need it, but please double-check ones you know about in case I've missed any. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
Hi all, The trees have reopened, based on the fact that the relevant tests are passing again reliably. That doesn't mean the work is over. Remaining things I'm aware of: - Finish fixing the devtools leaks: https://bugzilla.mozilla.org/show_bug.cgi?id=932880. One patch has landed, bgrinstead has another with r+ and has almost finished on try and should land soon. - Finish fixing the social API leaks: https://bugzilla.mozilla.org/show_bug.cgi?id=933551. Should be done by markh or mixedpuppy, AFAICT? - Improve the shutdown leak detection situation: https://bugzilla.mozilla.org/show_bug.cgi?id=932898. Tim Taubert is working on this, has patches in progress. It's possible that other leaks might be uncovered by this change, which would block it landing. If that happens, and you know are involved with such uncovered leaks, please fix them ASAP. We don't want to backslide. - Finish the post-mortem, in order to understand if we can do anything else to avoid problems like these in the future: https://etherpad.mozilla.org/mEB0H50ZjX. Please contribute! Also, I'm sure everyone has patches ready to land, but if we can be gentle on the trees that will reduce the likelihood of re-closure. Pace yourselves. Thanks! Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Closure of trunk trees - owners for bugs needed
On Wed, Oct 30, 2013 at 11:04 AM, Ed Morley wrote: > Hi all! > > Trunk trees are currently closed [1] - the requirements for reopening are: > https://bugzilla.mozilla.org/show_bug.cgi?id=932781#c11 I have (slightly optimistically) started writing a post-mortem of this closure, analyzing what went wrong and why, and how we might avoid it in the future: https://etherpad.mozilla.org/mEB0H50ZjX It's only partially written, because I only have a partial understanding. In particular, the Mochi-2 failures and fixes are well described, but the Mochi-bc failures and fixes are less so. Andrew McCreight has already added some good details, and I ask everyone else who has been involved with diagnosing and fixing this situation to do likewise. Thanks. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [Sheriffs] Closure of trunk trees - owners for bugs needed
Since the concern has already been raised on IRC a few times about the likelihood of bustage pileups once the tree re-opens, I would also like to also throw out a quick reminder that Try wait times are probably the lowest they will ever be during a regular week because of this closure, so *please* do make sure any patches you're waiting to push have a green run first. Feel free to ping a sheriff in #developers if you're not sure about one of the results you're seeing. Thanks! -Ryan - Original Message - From: "Carsten Book" To: "Ed Morley" Cc: "Sheriffs" , "dev.platform" , "dev.tree-management" Sent: Thursday, October 31, 2013 3:50:18 AM Subject: Re: [Sheriffs] Closure of trunk trees - owners for bugs needed Hi, good morning. Since the Trees are still closed here is a update as of 0:45am Pacific Time * https://bugzilla.mozilla.org/show_bug.cgi?id=932898 has an owner now * https://bugzilla.mozilla.org/show_bug.cgi?id=932880 Patch landed on inbound and tests are running. Cheers, - Tomcat - Original Message - From: "Ed Morley" To: "Sheriffs" , "dev.tree-management" , "dev.platform" Sent: Wednesday, October 30, 2013 7:04:45 PM Subject: [Sheriffs] Closure of trunk trees - owners for bugs needed Hi all! Trunk trees are currently closed [1] - the requirements for reopening are: https://bugzilla.mozilla.org/show_bug.cgi?id=932781#c11 tl;dr we need owners for these bugs: * https://bugzilla.mozilla.org/show_bug.cgi?id=932898 * https://bugzilla.mozilla.org/show_bug.cgi?id=932880 Please can anyone who has any spare cycles take a look! :-) Best wishes, Ed [1] Apart from b2g-inbound, since that doesn't run Windows 7 tests, which are the ones that are being the most problematic. On 30 October 2013 14:09:47, Ed Morley wrote: > I've broken the tree closing issues out into > https://bugzilla.mozilla.org/show_bug.cgi?id=932781 and pasted the > relevant parts of the IRC conversations from last night, because the > OOMs are not constrained to that one existing bug & it's good to get > the IRC chat saved somewhere visible (I've only just managed to piece > together what has been tried so far and where we're at). > > khuey is kindly looking at this some more once he's gotten sorted :-) > > Ed ___ Sheriffs mailing list sheri...@mozilla.org https://mail.mozilla.org/listinfo/sheriffs ___ Sheriffs mailing list sheri...@mozilla.org https://mail.mozilla.org/listinfo/sheriffs ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [Sheriffs] Closure of trunk trees - owners for bugs needed
Hi, good morning. Since the Trees are still closed here is a update as of 0:45am Pacific Time * https://bugzilla.mozilla.org/show_bug.cgi?id=932898 has an owner now * https://bugzilla.mozilla.org/show_bug.cgi?id=932880 Patch landed on inbound and tests are running. Cheers, - Tomcat - Original Message - From: "Ed Morley" To: "Sheriffs" , "dev.tree-management" , "dev.platform" Sent: Wednesday, October 30, 2013 7:04:45 PM Subject: [Sheriffs] Closure of trunk trees - owners for bugs needed Hi all! Trunk trees are currently closed [1] - the requirements for reopening are: https://bugzilla.mozilla.org/show_bug.cgi?id=932781#c11 tl;dr we need owners for these bugs: * https://bugzilla.mozilla.org/show_bug.cgi?id=932898 * https://bugzilla.mozilla.org/show_bug.cgi?id=932880 Please can anyone who has any spare cycles take a look! :-) Best wishes, Ed [1] Apart from b2g-inbound, since that doesn't run Windows 7 tests, which are the ones that are being the most problematic. On 30 October 2013 14:09:47, Ed Morley wrote: > I've broken the tree closing issues out into > https://bugzilla.mozilla.org/show_bug.cgi?id=932781 and pasted the > relevant parts of the IRC conversations from last night, because the > OOMs are not constrained to that one existing bug & it's good to get > the IRC chat saved somewhere visible (I've only just managed to piece > together what has been tried so far and where we're at). > > khuey is kindly looking at this some more once he's gotten sorted :-) > > Ed ___ Sheriffs mailing list sheri...@mozilla.org https://mail.mozilla.org/listinfo/sheriffs ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Closure of trunk trees - owners for bugs needed
Hi all! Trunk trees are currently closed [1] - the requirements for reopening are: https://bugzilla.mozilla.org/show_bug.cgi?id=932781#c11 tl;dr we need owners for these bugs: * https://bugzilla.mozilla.org/show_bug.cgi?id=932898 * https://bugzilla.mozilla.org/show_bug.cgi?id=932880 Please can anyone who has any spare cycles take a look! :-) Best wishes, Ed [1] Apart from b2g-inbound, since that doesn't run Windows 7 tests, which are the ones that are being the most problematic. On 30 October 2013 14:09:47, Ed Morley wrote: I've broken the tree closing issues out into https://bugzilla.mozilla.org/show_bug.cgi?id=932781 and pasted the relevant parts of the IRC conversations from last night, because the OOMs are not constrained to that one existing bug & it's good to get the IRC chat saved somewhere visible (I've only just managed to piece together what has been tried so far and where we're at). khuey is kindly looking at this some more once he's gotten sorted :-) Ed ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
