I was thinking of the challenge with sporadic/random failures the other day and what would help. I think more and smarter notifications of failures could help a lot.
(A) Using Git history, a Jenkins plugin could send an email to anyone who touched the failing test in the last 4 weeks. If that list is empty then choose the most recent person. This notification does not go to the dev list. Rationale: People who most recently maintained the test in some way are likely to want to help keep it passing. (B) (At fucit.com?) if a test has not failed in the 4 weeks prior, then notify the dev list with an email about just this test (in subject). If "many" tests fail in a build, then those failures don't count for this tracking. Rationale: Any active developer ought to take notice as this may be caused by one of their commits. Note: if "many" tests fail in a build, then it's likely a reproducible recently-committed change with a wide blast radius that is going to be fixed soon and which will already be reported by standard Jenkins notifications. These are just some ideas. I looked for a Jenkins plugin that did (A) but found none. It seems most build setups including ours aren't oriented around longitudinal tracking of individual tests, and are instead just overall pass/fail tracking of the entire suite. Hoss (& Mark?) have helped track tests longitudinally but it's a separate system that one must manually look at; it's not integrated with Jenkins nor with notifications. ~ David On Tue, Jul 31, 2018 at 3:00 AM Dawid Weiss <dawid.we...@gmail.com> wrote: > Hi Erick, > > > Is anybody paying the least attention to this or should I just stop > bothering? > > I think your effort is invaluable, although if not backed by actions > to fix those bugs > it's pointless. I'm paying attention to the Lucene part. As for Solr > tests I admit I gave > up hope a longer while ago. I can't run past Solr tests on my machine > anymore, no > matter how many runs I try. Yes, this means I commit stuff back if I > can run precommit > and Lucene tests only -- it is terrible, but a fact. > > > But with an additional 63 tests to BadApple [...] > > Exactly. I don't see this situation getting any better, even with all > your (and other people's) work > put into fixing them. I don't have any ideas or solution for this, I'm > afraid. > > Dawid > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.solrenterprisesearchserver.com
