On Tue, Apr 10, 2012 at 11:42 AM, Stephen Chenney <[email protected]>wrote:
> On Tue, Apr 10, 2012 at 1:00 PM, Ryosuke Niwa <[email protected]> wrote: > >> On Tue, Apr 10, 2012 at 6:10 AM, Stephen Chenney >> <[email protected]>wrote: >> >>> There is a significant practical problem to "turn the tree red and work >>> with someone to rebaseline the tests". It takes multiple hours for some >>> bots to build and test a given patch. That means, at any moment, you will >>> have maybe tens and in some cases hundreds of failing tests associated with >>> some changelist that you need to track on the bots. You might have more >>> failing tests associated with a different changelist, and so on. >> >> >> But you have to do this for non-Chromium ports anyway because they don't >> use test_expectations.txt and skipping the tests won't help you generate >> new baseline. In my opinion, we should not further diverge from the way >> things are done in other ports. >> > > How long on average does it take a builder to get through a change on > another port? Right now the Chromium Mac 10.5 and 10.6 dbg builds are > processing a patch from about 3 hours ago. About 20 patches have gone in > since then. For the Mac 10.5 tree to ever be green would require there > being no changes at all requiring new baselines for a 3 hour window. > > Just because other teams do it some way does not mean that Chromium, with > it's greater number of bots and platforms, should do it the same way. > Yes, it does mean that we should do it the same way. What if non-Chromium ports started imposing arbitrary processes like this on the rest of us? It'll be a total chaos, and nobody would understand the right thing to do for all ports. > We are discussing a process here, not code, and in my mind the goal is to > have the tree be as green as possible with all failures tracked with a > "minimal" expectations file and as little engineer time as possible. > That's not our project goal. We have continuous builds and regression tests to prevent regressions to improve the stability, not to keep bots green. Please review http://www.webkit.org/projects/goals.html Just look at how often the non-chromium mac and win builds are red. In > particular, changes submitted via the commit queue take an indeterminate > amount of time to go in, anything from an hour to several hours. Patch > authors do not necessarily even have control over when the CQ+ is given. > That's why I don't use commit queue when I know my patch requires platform-dependent rebaselines. > Even when manually committing, if it takes 3 hours to create baselines > then no patches go in in the afternoon. What if the bots are down or > misbehaving? > We need to promptly fix those bots. I would also point out the waste of resources when every contributor needs > to track every failure around commit time in order to know when their own > changes cause failures, and then track the bots to know when they are free > to go home. > But that's clearly stated in the contribution guide line. Why not simply attach an owner and a resolution date to each expectation? >>> The real problem right now is accountability and a way to remind people >>> that they have left expectations hanging. >>> >> >> That's what WebKit bugs are for. Ossy frequently files a bug and cc'es >> the patch author when a new test is added or a test starts failing and he >> doesn't know whether new result is correct or not. He also either skips the >> test or rebaseline the test as needed. He also reverts patches when the >> patch clearly introduced serious regressions (e.g. crashes on hundreds of >> tests). >> > > Yes, Ossy does an excellent job of gardening. Unfortunately, on Chrome we > have tens if not hundreds of gardeners and, as this thread has revealed, no > clear agreement on the best way to garden. > That IS the problem. We have too many in-experiented gardeners that don't understand the WebKit culture or the WebKit process. I strongly believe that keeping the tree green is more important than > having a clean expectations file. > I disagree. You're effectively just disabling the test temporarily. Finally, there is no pain free way to do this. The question is how to > distribute the pain. Right now each gardening is using a process that > distributes pain in their preferred way. From a community standpoint it > would be nice if the Chromium team could come up with something consistent. > The process Chromium port uses should be consistent with non-Chromium ports. - Ryosuke
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
