Updates:
Cc: maruelatchromium
Comment #4 on issue 21789 by [email protected]: Update test_expectations
automatically to match reality
http://code.google.com/p/chromium/issues/detail?id=21789
The more I think about this, the more I think the benefits do not outweigh
the costs.
Firstly, if we're going to automatically update test_expectations, then we
should also automatically file
bugs and email the people on the blamelist for when the test started
failing. This is mainly important so
that new failures get fixed when they get introduced and not weeks/months
later.
But, since we have a lot of flaky tests, this would be really noisy and
people would likely start ignoring
them or assuming someone else on the blamelist caused it. And at this
point, we're not really doing
much different than the tree closing bot would do.
I think we should instead focus our effort on eliminating flakiness in the
webkit tests. And then having
the tree auto close on regressions.
maruel: would be nice if we could start sending out the blame emails
without closing the tree for
webkit tests and UI tests even before all the flakiness is eliminated.
The only thing we lose here is updating test_expectations when a failing
test start passing. I would be
OK with automating the *removal* of outdated expectations. That's less work
and less likely to have
negative consequences.
Although really, a prerequisite to being able to do any of this would be
making the flakiness dashboard
persistent. Currently, if a webkit test bot is clobbered, we start with a
blank slate.
--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
--~--~---------~--~----~------------~-------~--~----~
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/group/chromium-bugs
-~----------~----~----~----~------~----~------~--~---
