On Thursday, May 30, 2013 3:23:50 AM UTC+2, Nicolas Pierron wrote: > I think the recent trouble was mostly that we got multiple regressions in > the one range of commits.
True and that is also the reason I want to re-empower people again when that happens. People shouldn't be afraid to backout regressions that have not been approved. Note that this is also a problem when the revisions are in visible different measurements on arewefastyet. It is a bit easier to distinguish, but still causes problems. Arewefastyet is not really stable, we have spikes and small differences between runs. If measurement (a) shows a 20% regression and (b) shows a 5% improvement. This could be the regression is in either (a) or (b) or both... In case of only one regression in most cases it is visible in the commit log. It is possible to pinpoint a revision, just by looking at the commit log. If both changelog contains possible revisions causing problems, then it is harder ... Backout could help here to make sure there is no regressions left or to pinpoint the second regression > I don't know if one computer has enough resources for that but it might be > interesting if the continuous integration system was able to bisect the > performance changes for us. Either by ignoring changes out-side the > JavaScript engine directory and scheduling one commit after the other. Or > by using the idle time to provide additional result when a non-noise bump is > observed in the results. That would be really cool. But I assume would require some manpower. I think you can ask Dave Anderson that although not much is needed on awfy, it takes a lot of time to maintain. Do we really want to take time away from adding optimization to have such a system, when good rules can get us already that far > One thing that I want to clarify is the importance of each platforms, in > addition to the importance of each benchmark. We should not ignore B2G, > even if it is 2 click away from the main page. It is indeed true that regressions on B2G and ARM are noticed later-on. But I don't think there is a different ruling about regression on those platforms than on x86. At least I have tried to treat them the same. Though it's a bit harder, because the changelog length is longer and two runs on the same revision tend to have bigger variation yand I don't have a B2G phone set-up (yet) to blame the right patch. So I don't think people don't agree on these ruling also applying to B2G and ARM. But more that people aren't actively looking/watching those trees and/or that it is harder to watch. So (if that wasn't clear already), these policies/rules/guidelines indeed apply on x86 and B2G !!! > -- > > Nicolas B. Pierron _______________________________________________ dev-tech-js-engine-internals mailing list [email protected] https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
