On Fri, Aug 21, 2009 at 2:33 PM, Pam Greene <[email protected]> wrote:
> At least in the batch of tests I examined, the ones that needed > re-baselining weren't tests we'd originally failed and suddenly started > passing. They were new tests that nobody had ever taken a good look at. > > If that matches everyone else's experience, then all we need is an > UNTRIAGED annotation in the test_expectations file to mark ones the next > Great Re-Baselining needs to examine. > What happens when someone forgets to set this flag? Didn't we want to avoid adding any more state to the test_expectations file? On Fri, Aug 21, 2009 at 1:47 PM, Evan Martin <[email protected]> wrote: > > This seems to me like a lot more work for minimal gain. Because > you've thought more about it than I have, it makes me think I'm > misunderstanding something. Can you explain this more simply, in > terms of use cases? > > Here's what I think you're saying: > 1) We don't have notes on why tests are failing. => Why not annotate > the tests in test_lists? That's what I've always done. Once again, we don't want to add more state to the test_expectations. How may people looked up the tests they were supposed to rebaseline in this file to see if there were notes? I kind of doubt anyone. > 2) We don't have a way of tracking when a failing test output changes. > => But failing is failing; no matter what you want a human to look > at the result before you mark it as "passing", so it doesn't seem like > it's worth a bunch of extra machinery to track this. And if a test > starts passing it gets marked "unexpected pass" by the builders > already, and it also seems like a human should look at it. There are different reasons for failing. A layout test could be failing because of a known bug and then start failing in a different way (later) due to a regression. When a bug fails in a new way, it's worth taking a quick look, I think. On Fri, Aug 21, 2009 at 7:52 PM, Dirk Pranke <[email protected]> wrote: > On Fri, Aug 21, 2009 at 6:43 PM, Ojan Vafai<[email protected]> wrote: > > On Fri, Aug 21, 2009 at 4:54 PM, Peter Kasting <[email protected]> > > wrote: > >> > >> On Fri, Aug 21, 2009 at 4:50 PM, Dirk Pranke <[email protected]> > wrote: > >>> > >>> This is all good feedback, thanks! To clarify, though: what do you > >>> think the cost will be? Perhaps you are assuming things about how I > >>> would implement this that are different than what I had in mind. > >> > >> Some amount of your time, and some amount of space on the bots. > > > > Also, some amount of the rest of the team's time to follow this process. > > Ojan > > Okay, it sounds like there's enough initial skepticism that it's > probably worth doing a hack before pushing this fully through. I think > I'll try to take a few snapshots of the layout test failures over a > few days and see if we see any real diffs, and then report back. > All of this said, I agree that there is a cost to maintaining this that I didn't consider at first. I think the approach you're taking Dirk (doing it locally for a while and seeing if it's useful) is probably the right one. Of course the long term solution is to get the layout test failures to 0. :-) J --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
