Wow that HBase UI looks super useful. +1 to having something like that. If not, +1 to having a proper nightly build, it would help devs identify which commits break which tests. I find using git-bisect can take a long time to run, and can be difficult to use (e.g. finding a known good commit isn't always easy).
On Mon, Mar 5, 2018 at 9:03 AM, Peter Vary <pv...@cloudera.com> wrote: > Without a nightly build and with this many flaky tests it is very hard to > identify the braking commits. We can use something like bisect and multiple > test runs. > > There is a more elegant way to do this with nightly test runs: > https://issues.apache.org/jira/browse/HBASE-15917 < > https://issues.apache.org/jira/browse/HBASE-15917> > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/ > lastSuccessfulBuild/artifact/dashboard.html <https://builds.apache.org/ > job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html> > > This also helps to identify the flaky tests, and creates a continuos, > updated list of them. > > > On Feb 23, 2018, at 6:55 PM, Sahil Takiar <takiar.sa...@gmail.com> > wrote: > > > > +1 > > > > Does anyone have suggestions about how to efficiently identify which > commit > > is breaking a test? Is it just git-bisect or is there an easier way? Hive > > QA isn't always that helpful, it will say a test is failing for the past > > "x" builds, but that doesn't help much since Hive QA isn't a nightly > build. > > > > On Thu, Feb 22, 2018 at 10:31 AM, Vihang Karajgaonkar < > vih...@cloudera.com> > > wrote: > > > >> +1 > >> Commenting on JIRA and giving a 24hr heads-up (excluding weekends) > would be > >> good. > >> > >> On Thu, Feb 22, 2018 at 10:19 AM, Alan Gates <alanfga...@gmail.com> > wrote: > >> > >>> +1. > >>> > >>> Alan. > >>> > >>> On Thu, Feb 22, 2018 at 8:25 AM, Thejas Nair <thejas.n...@gmail.com> > >>> wrote: > >>> > >>>> +1 > >>>> I agree, this makes sense. The number of failures keeps increasing. > >>>> A 24 hour heads up in either case before revert would be good. > >>>> > >>>> > >>>> On Thu, Feb 22, 2018 at 2:45 AM, Peter Vary <pv...@cloudera.com> > >> wrote: > >>>> > >>>>> I agree with Zoltan. The continuously braking tests make it very hard > >>> to > >>>>> spot real issues. > >>>>> Any thoughts on doing it automatically? > >>>>> > >>>>>> On Feb 22, 2018, at 10:47 AM, Zoltan Haindrich <k...@rxd.hu> > >> wrote: > >>>>>> > >>>>>> * > >>>>>> > >>>>>> Hello, > >>>>>> > >>>>>> * > >>>>>> * > >>>>>> > >>>>>> ** > >>>>>> > >>>>>> In the last couple weeks the number of broken tests have started to > >>> go > >>>>> up...and even tho I run bisect/etc from time to time ; sometimes > >> people > >>>>> don’t react to my comments/tickets/etc. > >>>>>> > >>>>>> Because keeping this many failing tests makes it easier for a new > >> one > >>>> to > >>>>> slip in...I think reverting the patch introducing the test failures > >>> would > >>>>> also help in some case. > >>>>>> > >>>>>> I think it would help a lot to prevent further test breaks to > >> revert > >>>> the > >>>>> patch if any of the following conditions is met: > >>>>>> > >>>>>> * > >>>>>> * > >>>>>> > >>>>>> C1) if the notification/comment about the fact that the patch > >> indeed > >>>>> broken a test somehow have been unanswered for at least 24 hours. > >>>>>> > >>>>>> C2) if the patch is in for 7 days; but the test failure is still > >> not > >>>>> addressed (note that in this case there might be a conversation about > >>>>> fixing it...but in this case ; to enable other people to work in a > >>>> cleaner > >>>>> environment is more important than a single patch - and if it can't > >> be > >>>>> fixed in 7 days...well it might not get fixed in a month). > >>>>>> > >>>>>> * > >>>>>> * > >>>>>> > >>>>>> I would like to also note that I've seen a few tickets which have > >>> been > >>>>> picked up by people who were not involved in creating the original > >>>> change - > >>>>> and although the intention was good, they might miss the context of > >> the > >>>>> original patch and may "fix" the tests in the wrong way: accept a > >> q.out > >>>>> which is inappropriate or ignore the test... > >>>>>> > >>>>>> * > >>>>>> * > >>>>>> > >>>>>> would it be ok to implement this from now on? because it makes my > >>>>> efforts practically useless if people are not reacting… > >>>>>> > >>>>>> * > >>>>>> * > >>>>>> > >>>>>> note: just to be on the same page - this is only about running a > >>> single > >>>>> test which falls on its own - I feel that flaky tests are an entirely > >>>>> different topic. > >>>>>> > >>>>>> * > >>>>>> * > >>>>>> > >>>>>> cheers, > >>>>>> > >>>>>> Zoltan > >>>>>> > >>>>>> ** > >>>>>> * > >>>>> > >>>>> > >>>> > >>> > >> > > > > > > > > -- > > Sahil Takiar > > Software Engineer > > takiar.sa...@gmail.com | (510) 673-0309 > > -- Sahil Takiar Software Engineer takiar.sa...@gmail.com | (510) 673-0309
