Re: Test failures are out of control......

You can also increase the suite timeout for a particular test with @TimeoutSuite(...) (although making it too long doesn't seem right too, even nightlies should terminate in a sensible time). D. On Tue, Feb 27, 2018 at 11:37 PM, Karl Wright wrote: > It succeeds but it takes

Re: Test failures are out of control......

It succeeds but it takes too long, so I committed a reduction in the number of iterations that are attempted for nightly from 20 to 5. That should make it pass reliably. Karl On Tue, Feb 27, 2018 at 3:52 PM, Erick Erickson wrote: > Karl: > > Let us know if

Re: Test failures are out of control......

Karl: Let us know if you can't get this to fail locally, we can turn it back on on Jenkins as long as it's getting active attention. Erick On Tue, Feb 27, 2018 at 12:40 PM, Karl Wright wrote: > I looked at the Geo3d failures; these are timeouts exclusively. I suspect >

Re: Test failures are out of control......

I looked at the Geo3d failures; these are timeouts exclusively. I suspect we're seeing a test issue. Will try to correct. Karl On Tue, Feb 27, 2018 at 1:51 PM, Chris Hostetter wrote: > : The most interesting bit is probably here... > : > :

Re: Test failures are out of control......

: The most interesting bit is probably here... : : http://fucit.org/solr-jenkins-reports/failure-report.html FYI: I realized this morning that the "Suite Runs" counts were being artificially inflated for suites that are frequently SKIPPED (either because they are @Nighly or @Slow and not

Re: Test failures are out of control......

That's fine. I'm not totally clear what the "anti-regression" path forward is. This should make tests less flakey, right? I'd guess that if we test with badapples=true and don't get failures for a while for some tests we'll try un-BadAppling tests as time passes. Erick P.S. Besides,it's already

Re: Test failures are out of control......

TLDR; I'm going to push https://issues.apache.org/jira/browse/SOLR-12027 in a day. Let me know if you think it's a bad idea. On Fri, Feb 23, 2018 at 8:06 PM, Erick Erickson wrote: > Testing distributed systems requires, well, distributed systems which > is what starting

Re: Test failures are out of control......

Testing distributed systems requires, well, distributed systems which is what starting clusters is all about. The great leap of faith of individual-method unit testing is that if all the small parts are tested, combining them in various ways will "just work". This is emphatically not true with

Re: Test failures are out of control......

> Randomness makes it difficult to correlate a failure to the commit that made > the test to fail (as was pointed out earlier in the discussion). If each > execution path is different, it may very well be that a failure you > experience is introduced several commits ago, so it may not be your

Re: Test failures are out of control......

0 > Subject: Re: Test failures are out of control.. > To: dev@lucene.apache.org > > > See: https://issues.apache.org/jira/browse/SOLR-12016 > > And there are already several linked issues that have been fixed, I'll > try removing the annotations in SOLR-12017. > > E

Re: Test failures are out of control......

See: https://issues.apache.org/jira/browse/SOLR-12016 And there are already several linked issues that have been fixed, I'll try removing the annotations in SOLR-12017. Erick On Thu, Feb 22, 2018 at 9:55 AM, Uwe Schindler wrote: >> : great and very helpful! Does it only

RE: Test failures are out of control......

> : great and very helpful! Does it only contain Solr or are there also Lucene > tests reported? > > That was just a dumb choice on my part when creating the fucit.org URL > since the vast majority of the jenkins failures are in Solr tests. > > It's really every test failure reported from any

RE: Test failures are out of control......

ter [mailto:hossman_luc...@fucit.org] : > Sent: Thursday, February 22, 2018 1:56 AM : > To: dev@lucene.apache.org : > Subject: Re: Test failures are out of control.. : > : > : > : * Hoss has worked on aggregating all test failures from the 3 Jenkins : > : systems (ASF, Poli

Re: Test failures are out of control......

On 2/22/2018 9:04 AM, Erick Erickson wrote: Thanks all for contributing to the discussion. I'll write up a JIRA before Monday and try to summarize the discussion and we can go from there. Hmmm, a LUCENE or SOLR JIRA? Does it matter? That's a head scratcher.  The majority of what makes the

Re: Test failures are out of control......

Thanks all for contributing to the discussion. I'll write up a JIRA before Monday and try to summarize the discussion and we can go from there. Hmmm, a LUCENE or SOLR JIRA? Does it matter? One thing I also realized is that my use-case would be served just fine with an easy way to identify runs

Re: Test failures are out of control......

On Thu, Feb 22, 2018 at 8:59 AM, Adrien Grand wrote: > I understand your point Yonik, but the practical consequences are worse I think that's what we were debating though. IMO, the overall practical consequences of simply sweeping the problem under the rug by disabling all

Re: Test failures are out of control......

+1 Dawid I understand your point Yonik, but the practical consequences are worse than disabling these tests like Erick pointed out in his initial emails. If we are concerned about forgetting these disabled tests, which is a concern I agree, I think Uwe's idea to add a weekly job that runs with

RE: Test failures are out of control......

it.org] > Sent: Thursday, February 22, 2018 1:56 AM > To: dev@lucene.apache.org > Subject: Re: Test failures are out of control.. > > > : * Hoss has worked on aggregating all test failures from the 3 Jenkins > : systems (ASF, Policeman, and Steve's), downloading the test resu

Re: Test failures are out of control......

Don't know, Yonik... If I make a change I am interested in regressions from the start state; running flaky tests makes it impossible and frustrating (and pointless in my opinion). I don't think my expectations are that much off from the average - you may wake up being the only person who has the

Re: Test failures are out of control......

: * Hoss has worked on aggregating all test failures from the 3 Jenkins : systems (ASF, Policeman, and Steve's), downloading the test results & logs, : and running some reports/stats on failures. He should be ready to share : this more publicly soon. I think Steve's linked to some of this before

RE: Test failures are out of control......

...@thetaphi.de > -Original Message- > From: Yonik Seeley [mailto:ysee...@gmail.com] > Sent: Thursday, February 22, 2018 12:17 AM > To: Solr/Lucene Dev <dev@lucene.apache.org> > Subject: Re: Test failures are out of control.. > > On Wed, Feb 21, 2018 a

Re: Test failures are out of control......

On Wed, Feb 21, 2018 at 6:13 PM, Uwe Schindler wrote: >> > There are exactly three >> > BadApple annotations in the entire code base at present, is there >> > enough value in introducing another annotation to make it worthwhile? >> >> If we change BadApple tests to be executed

RE: Test failures are out of control......

> > There are exactly three > > BadApple annotations in the entire code base at present, is there > > enough value in introducing another annotation to make it worthwhile? > > If we change BadApple tests to be executed by default for "ant test" > (but not for the most frequent jenkins jobs), then

Re: Test failures are out of control......

This issue is hugely important. At Lucidworks we have implemented a "Test Confidence" role that focuses on improving the ability of all members of the community to trust that reported failures from any of the Jenkins systems are actual failures and not flakey tests. This role rotates among the

Re: Test failures are out of control......

On Wed, Feb 21, 2018 at 5:52 PM, Erick Erickson wrote: > There are exactly three > BadApple annotations in the entire code base at present, is there > enough value in introducing another annotation to make it worthwhile? If we change BadApple tests to be executed by

Re: Test failures are out of control......

I don't have strong opinions about what we do with our existing flaky tests. I think re-running failures before commit might theoretically catch more bugs than ignoring the test outright, but with all the noise and how standard it is to need to rerun tests I'd be surprised if the numbers are all

RE: Test failures are out of control......

Hi, > Flakey Test Problems: > a) Flakey tests create so much noise that people no longer pay > attention to the automated reporting via email. > b) When running unit tests manually before a commit (i.e. "ant test") > a flakey test can fail. > > Solutions: > We cloud fix (a) by marking as flakey

Re: Test failures are out of control......

Yonik: Good discussion. I'm not wedded to a particular solution, it's just the current direction is not sustainable. I'll back up a bit and see if I can state my goals more clearly, it looks like we're arguing for much the same thing. > I want e-mail messages with test failures to be worth

Re: Test failures are out of control......

On Wed, Feb 21, 2018 at 3:26 PM, Erick Erickson wrote: > Yonik: > > What I'm frustrated by now is that variations on these themes haven't > cured the problem, and it's spun out of control and is getting worse. I understand, but what problem(s) are you trying to solve?

Re: Test failures are out of control......

Yonik: What I'm frustrated by now is that variations on these themes haven't cured the problem, and it's spun out of control and is getting worse. It's the "getting worse" part that is most disturbing. Continuing as we have in the past isn't working, it's time to try something else. There are 17

Re: Test failures are out of control......

We should be careful not to conflate running of unit tests with automated reporting, and the differing roles that flakey tests play in different scenarios. For example, I no longer pay attention to automated failure reports, esp if I haven't committed anything recently. However, when I'm making

Re: Test failures are out of control......

Dawid: Yep, definitely a recurring theme. But this time I may actually, you know, do something about it ;) Mark is one of the advocates of this theme, perhaps he got exhausted trying to push that stone up the hill ;). Maybe it's my turn to pick up the baton Comments about there being value to

Re: Test failures are out of control......

It's a recurring theme, huh, Erick? :) http://markmail.org/message/7eykbuyyaxbxn364 I agree with your opinion and I have expressed it more than once -- a test that is failing for longer while and cannot be identified or fixed is a candidate for removal. The noise in Solr tests have increased to

Re: Test failures are out of control......

+1, agree with Adrien, thanks for bringing this up Erick! Il giorno mer 21 feb 2018 alle ore 17:15 Adrien Grand ha scritto: > Thanks for bringing this up Erick. I agree with you we should silence > those frequent failures. Like you said, the side-effects of not silencing >

Re: Test failures are out of control......

Thanks for bringing this up Erick. I agree with you we should silence those frequent failures. Like you said, the side-effects of not silencing them are even worse. I'll add that these flaky tests also make releasing harder, it took me three runs last time (Lucene/Solr 7.2) for the release build

Test failures are out of control......

There's an elephant in the room, and it's that failing tests are being ignored. Mind you, Solr and Lucene are progressing at a furious pace with lots of great functionality being added. That said, we're building up a considerable "technical debt" when it comes to testing. And I should say up