Yonik: What I'm frustrated by now is that variations on these themes haven't cured the problem, and it's spun out of control and is getting worse. It's the "getting worse" part that is most disturbing. Continuing as we have in the past isn't working, it's time to try something else.
There are 17 open JIRAs for tests right now. Some as far back as 2010, listed below. Since last night I've collected 49 distinct failures from the dev e-mails (haven't triaged them completely to see if some are contained in others, but "sort < all_of_them | uniq > my_file" results in a file 49 lines long). What I'm after here is a way to keep from backsliding further, and a path to getting better. That's what's behind the straw-man proposal that we get the tests to be clean, even if that means disabling the ones that are flakey. I should have emphasized more strongly that the corollary to disabling flakey tests is that we need to get aggressive about not tolerating _new_ flaky tests. I'm volunteering (I guess) to be the enforcer here, as much as public comments can be construed to be "enforcing" ;) If someone has a favorite test that they think adds value even if it fails fairly frequently, we can un-BadApple it provided someone is actively working on it. Or it can be un-BadAppled locally and/or temporarily. I'm perfectly fine with flakey tests being run and reported _if_ that helps resolve it. Also I'm volunteering to produce a "weekly BadApple" list so people can work on them as they see fit expressly to keep them from getting lost. (1) I have no idea how to do this, or even if it's possible. What do you have in mind? (2) doesn't seem to be working based on the open JIRAs below and the number of failing tests that are accumulating. (3a) Well, the first part is what my "enforcing" comment is about above I think ;) (3b) I'd argue that the part about "without losing test coverage" is partly addressed by the notion of running periodically with BadApple disabled. Which each individual can also do at their discretion if they care to. More importantly though, the test coverage isn't very useful when failures are ignored anyway. (4) I thoroughly applaud that one long term, but I'll settle in the short term for doing something to keep even _more_ from accumulating. I should also emphasize that disabling tests is certainly _NOT_ my preference, fixing them all is. I doubt that declaring a moratorium on all commits until all the tests were fixed is practical though ;) And without something changing in our approach, I don't see much progress being made. SOLR-10053 SOLR-10070 SOLR-10071 SOLR-10139 SOLR-10287 SOLR-10815 SOLR-11911 SOLR-2175 SOLR-4147 SOLR-5880 SOLR-6423 SOLR-6944 SOLR-6961 SOLR-6974 SOLR-8122 SOLR-8182 SOLR-9869 On Wed, Feb 21, 2018 at 11:12 AM, Yonik Seeley <[email protected]> wrote: > We should be careful not to conflate running of unit tests with > automated reporting, and the differing roles that flakey tests play in > different scenarios. > For example, I no longer pay attention to automated failure reports, > esp if I haven't committed anything recently. > However, when I'm making code changes and do "ant test", I certainly > pay attention to failures and re-run any failing tests. It sucks to > have to re-run a test just because it's flakey, but it's better than > accidentally committing a bug because test coverage was reduced. > > I'd suggest: > 1) fix/tweak automated test reporting to increase relevancy for developers > 2) open a JIRA for each flakey test and evaluate impact of removal on > test coverage > 3) If a new feature is added, and the test turns out to be flakey, > then the feature itself should be disabled before release. This > prevents both new flakey tests without resulting in loss of test > coverage, as well as motivates those who care about the feature to fix > the tests. > 4) fix flakey tests ;-) > > -Yonik > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
