Not adding badapples this week.
With the long weekend and the fact that the number of non-BadApple tests is fairly small this week, I'll skip adding more BadApple tests until next week. We're scarily close to the non-BaApple'd tests coming under control. If we're lucky, we can draw a line in the sand soon then start working on the backlog. I'll be encouraged if we can start shrinking the BadApple'd tests. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Not adding badapples this week.
Thanks for helping on this front Erick. I noticed a significant decrease in noise since you started badapple-ing bad tests, but I'm observing that our smoke-release builds still keep failing because of Solr tests (10 out of the last 10 builds) in spite of the fact that they disable bad apples: - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ Do you know whether there is something that makes these jobs any different? Le mar. 29 mai 2018 à 18:12, Erick Erickson a écrit : > With the long weekend and the fact that the number of non-BadApple > tests is fairly small this week, I'll skip adding more BadApple tests > until next week. > > We're scarily close to the non-BaApple'd tests coming under control. > If we're lucky, we can draw a line in the sand soon then start working > on the backlog. I'll be encouraged if we can start shrinking the > BadApple'd tests. > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: Not adding badapples this week.
Adrien: "Do you know whether there is something that makes these jobs any different?" Unfortunately no. I'm not really very well versed in the various test environments, maybe Uwe or Steve Rowe or Hoss might have some insight? On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand wrote: > Thanks for helping on this front Erick. I noticed a significant decrease in > noise since you started badapple-ing bad tests, but I'm observing that our > smoke-release builds still keep failing because of Solr tests (10 out of the > last 10 builds) in spite of the fact that they disable bad apples: > - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ > - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ > > Do you know whether there is something that makes these jobs any different? > > Le mar. 29 mai 2018 à 18:12, Erick Erickson a > écrit : >> >> With the long weekend and the fact that the number of non-BadApple >> tests is fairly small this week, I'll skip adding more BadApple tests >> until next week. >> >> We're scarily close to the non-BaApple'd tests coming under control. >> If we're lucky, we can draw a line in the sand soon then start working >> on the backlog. I'll be encouraged if we can start shrinking the >> BadApple'd tests. >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Not adding badapples this week.
I looked at the way that tests are run, an tthe only difference I see in the smoke tester jobs is that tests are run twice, once each for Java8 and Java9. Compared to non-smoke-tester jobs, this will double the likelihood of overall failure. I looked at the suites that failed in the last ten runs on those two smoke tester jobs. Except for SearchHandlerTest, which I have since (hopefully) fixed, there are seven suites with failed tests - here they are along with their packages: TestExecutePlanAction o.a.s.cloud.autoscaling.sim TestComputePlanAction o.a.s.cloud.autoscaling.sim TestTriggerIntegrationo.a.s.cloud.autoscaling.sim IndexSizeTriggerTest o.a.s.cloud.autoscaling CreateRoutedAliasTest o.a.s.cloud ReplaceNodeTest o.a.s.cloud MetricsHistoryHandlerTest o.a.s.handler.admin Some of those are pretty regular offenders AFAICT from http://fucit.org/solr-jenkins-reports/failure-report.html . Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and un-bad-apple’d its tests, but at least one of them is still failing since then - I’ll go add a comment on the issue. -- Steve www.lucidworks.com > On Jun 4, 2018, at 11:16 AM, Erick Erickson wrote: > > Adrien: > > "Do you know whether there is something that makes these jobs any different?" > > Unfortunately no. I'm not really very well versed in the various test > environments, maybe Uwe or Steve Rowe or Hoss might have some insight? > > On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand wrote: >> Thanks for helping on this front Erick. I noticed a significant decrease in >> noise since you started badapple-ing bad tests, but I'm observing that our >> smoke-release builds still keep failing because of Solr tests (10 out of the >> last 10 builds) in spite of the fact that they disable bad apples: >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ >> >> Do you know whether there is something that makes these jobs any different? >> >> Le mar. 29 mai 2018 à 18:12, Erick Erickson a >> écrit : >>> >>> With the long weekend and the fact that the number of non-BadApple >>> tests is fairly small this week, I'll skip adding more BadApple tests >>> until next week. >>> >>> We're scarily close to the non-BaApple'd tests coming under control. >>> If we're lucky, we can draw a line in the sand soon then start working >>> on the backlog. I'll be encouraged if we can start shrinking the >>> BadApple'd tests. >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > Begin forwarded message: > From: Erick Erickson > Subject: Re: Not adding badapples this week. > Date: June 4, 2018 at 11:16:56 AM EDT > To: dev@lucene.apache.org > Reply-To: dev@lucene.apache.org > > Adrien: > > "Do you know whether there is something that makes these jobs any different?" > > Unfortunately no. I'm not really very well versed in the various test > environments, maybe Uwe or Steve Rowe or Hoss might have some insight? > > On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand wrote: >> Thanks for helping on this front Erick. I noticed a significant decrease in >> noise since you started badapple-ing bad tests, but I'm observing that our >> smoke-release builds still keep failing because of Solr tests (10 out of the >> last 10 builds) in spite of the fact that they disable bad apples: >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ >> >> Do you know whether there is something that makes these jobs any different? >> >> Le mar. 29 mai 2018 à 18:12, Erick Erickson a >> écrit : >>> >>> With the long weekend and the fact that the number of non-BadApple >>> tests is fairly small this week, I'll skip adding more BadApple tests >>> until next week. >>> >>> We're scarily close to the non-BaApple'd tests coming under control. >>> If we're lucky, we can draw a line in the sand soon then start working >>> on the backlog. I'll be encouraged if we can start shrinking the >>> BadApple
Re: Not adding badapples this week.
Thanks Steve. IndexSizeTriggerTest seems to be the test that fails most often on this build indeed. Le lun. 4 juin 2018 à 19:30, Steve Rowe a écrit : > I looked at the way that tests are run, an tthe only difference I see in > the smoke tester jobs is that tests are run twice, once each for Java8 and > Java9. Compared to non-smoke-tester jobs, this will double the likelihood > of overall failure. > > I looked at the suites that failed in the last ten runs on those two smoke > tester jobs. Except for SearchHandlerTest, which I have since (hopefully) > fixed, there are seven suites with failed tests - here they are along with > their packages: > > TestExecutePlanAction o.a.s.cloud.autoscaling.sim > TestComputePlanAction o.a.s.cloud.autoscaling.sim > TestTriggerIntegrationo.a.s.cloud.autoscaling.sim > IndexSizeTriggerTest o.a.s.cloud.autoscaling > CreateRoutedAliasTest o.a.s.cloud > ReplaceNodeTest o.a.s.cloud > MetricsHistoryHandlerTest o.a.s.handler.admin > > Some of those are pretty regular offenders AFAICT from > http://fucit.org/solr-jenkins-reports/failure-report.html . > > Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and > un-bad-apple’d its tests, but at least one of them is still failing since > then - I’ll go add a comment on the issue. > > -- > Steve > www.lucidworks.com > > > On Jun 4, 2018, at 11:16 AM, Erick Erickson > wrote: > > > > Adrien: > > > > "Do you know whether there is something that makes these jobs any > different?" > > > > Unfortunately no. I'm not really very well versed in the various test > > environments, maybe Uwe or Steve Rowe or Hoss might have some insight? > > > > On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand wrote: > >> Thanks for helping on this front Erick. I noticed a significant > decrease in > >> noise since you started badapple-ing bad tests, but I'm observing that > our > >> smoke-release builds still keep failing because of Solr tests (10 out > of the > >> last 10 builds) in spite of the fact that they disable bad apples: > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ > >> > >> Do you know whether there is something that makes these jobs any > different? > >> > >> Le mar. 29 mai 2018 à 18:12, Erick Erickson a > >> écrit : > >>> > >>> With the long weekend and the fact that the number of non-BadApple > >>> tests is fairly small this week, I'll skip adding more BadApple tests > >>> until next week. > >>> > >>> We're scarily close to the non-BaApple'd tests coming under control. > >>> If we're lucky, we can draw a line in the sand soon then start working > >>> on the backlog. I'll be encouraged if we can start shrinking the > >>> BadApple'd tests. > >>> > >>> ------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >>> For additional commands, e-mail: dev-h...@lucene.apache.org > >>> > >> > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > Begin forwarded message: > > > From: Erick Erickson > > Subject: Re: Not adding badapples this week. > > Date: June 4, 2018 at 11:16:56 AM EDT > > To: dev@lucene.apache.org > > Reply-To: dev@lucene.apache.org > > > > Adrien: > > > > "Do you know whether there is something that makes these jobs any > different?" > > > > Unfortunately no. I'm not really very well versed in the various test > > environments, maybe Uwe or Steve Rowe or Hoss might have some insight? > > > > On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand wrote: > >> Thanks for helping on this front Erick. I noticed a significant > decrease in > >> noise since you started badapple-ing bad tests, but I'm observing that > our > >> smoke-release builds still keep failing because of Solr tests (10 out > of the > >> last 10 builds) in spite of the fact that they disable bad apples: > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ > >> > >> Do you know whether there is something
Re: Not adding badapples this week.
Yesterday I committed several fixes to the simulation package, which should improve reliability of these tests. If these failures still persist we should BadApple the ones that keep failing. > On 5 Jun 2018, at 10:14, Adrien Grand wrote: > > Thanks Steve. IndexSizeTriggerTest seems to be the test that fails most often > on this build indeed. > > Le lun. 4 juin 2018 à 19:30, Steve Rowe <mailto:sar...@gmail.com>> a écrit : > I looked at the way that tests are run, an tthe only difference I see in the > smoke tester jobs is that tests are run twice, once each for Java8 and Java9. > Compared to non-smoke-tester jobs, this will double the likelihood of > overall failure. > > I looked at the suites that failed in the last ten runs on those two smoke > tester jobs. Except for SearchHandlerTest, which I have since (hopefully) > fixed, there are seven suites with failed tests - here they are along with > their packages: > > TestExecutePlanAction o.a.s.cloud.autoscaling.sim > TestComputePlanAction o.a.s.cloud.autoscaling.sim > TestTriggerIntegrationo.a.s.cloud.autoscaling.sim > IndexSizeTriggerTest o.a.s.cloud.autoscaling > CreateRoutedAliasTest o.a.s.cloud > ReplaceNodeTest o.a.s.cloud > MetricsHistoryHandlerTest o.a.s.handler.admin > > Some of those are pretty regular offenders AFAICT from > http://fucit.org/solr-jenkins-reports/failure-report.html > <http://fucit.org/solr-jenkins-reports/failure-report.html> . > > Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and > un-bad-apple’d its tests, but at least one of them is still failing since > then - I’ll go add a comment on the issue. > > -- > Steve > www.lucidworks.com <http://www.lucidworks.com/> > > > On Jun 4, 2018, at 11:16 AM, Erick Erickson > <mailto:erickerick...@gmail.com>> wrote: > > > > Adrien: > > > > "Do you know whether there is something that makes these jobs any > > different?" > > > > Unfortunately no. I'm not really very well versed in the various test > > environments, maybe Uwe or Steve Rowe or Hoss might have some insight? > > > > On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand > <mailto:jpou...@gmail.com>> wrote: > >> Thanks for helping on this front Erick. I noticed a significant decrease in > >> noise since you started badapple-ing bad tests, but I'm observing that our > >> smoke-release builds still keep failing because of Solr tests (10 out of > >> the > >> last 10 builds) in spite of the fact that they disable bad apples: > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ > >> <https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/> > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ > >> <https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/> > >> > >> Do you know whether there is something that makes these jobs any different? > >> > >> Le mar. 29 mai 2018 à 18:12, Erick Erickson >> <mailto:erickerick...@gmail.com>> a > >> écrit : > >>> > >>> With the long weekend and the fact that the number of non-BadApple > >>> tests is fairly small this week, I'll skip adding more BadApple tests > >>> until next week. > >>> > >>> We're scarily close to the non-BaApple'd tests coming under control. > >>> If we're lucky, we can draw a line in the sand soon then start working > >>> on the backlog. I'll be encouraged if we can start shrinking the > >>> BadApple'd tests. > >>> > >>> - > >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >>> <mailto:dev-unsubscr...@lucene.apache.org> > >>> For additional commands, e-mail: dev-h...@lucene.apache.org > >>> <mailto:dev-h...@lucene.apache.org> > >>> > >> > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > <mailto:dev-unsubscr...@lucene.apache.org> > > For additional commands, e-mail: dev-h...@lucene.apache.org > > <mailto:dev-h...@lucene.apache.org> > > > > > > Begin forwarded message: > > > From: Erick Erickson > <mailto:erickerick...@gmail.com>> > > Subject: Re: Not adding badapples this week. > > Date: June 4, 2018 at 11:16:56 AM EDT > > To: dev
Re: Not adding badapples this week.
I’ll take the blame for that one as I said I’d take a look but never got to it. I should get some time during my flight for a conference tomorrow (after long) and I’ll make sure I prioritize this right after my talk prep. Should have enough time for both. Sorry about the noise! Anshum > On Jun 5, 2018, at 1:14 AM, Adrien Grand wrote: > > Thanks Steve. IndexSizeTriggerTest seems to be the test that fails most often > on this build indeed. > > Le lun. 4 juin 2018 à 19:30, Steve Rowe <mailto:sar...@gmail.com>> a écrit : > I looked at the way that tests are run, an tthe only difference I see in the > smoke tester jobs is that tests are run twice, once each for Java8 and Java9. > Compared to non-smoke-tester jobs, this will double the likelihood of > overall failure. > > I looked at the suites that failed in the last ten runs on those two smoke > tester jobs. Except for SearchHandlerTest, which I have since (hopefully) > fixed, there are seven suites with failed tests - here they are along with > their packages: > > TestExecutePlanAction o.a.s.cloud.autoscaling.sim > TestComputePlanAction o.a.s.cloud.autoscaling.sim > TestTriggerIntegrationo.a.s.cloud.autoscaling.sim > IndexSizeTriggerTest o.a.s.cloud.autoscaling > CreateRoutedAliasTest o.a.s.cloud > ReplaceNodeTest o.a.s.cloud > MetricsHistoryHandlerTest o.a.s.handler.admin > > Some of those are pretty regular offenders AFAICT from > http://fucit.org/solr-jenkins-reports/failure-report.html > <http://fucit.org/solr-jenkins-reports/failure-report.html> . > > Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and > un-bad-apple’d its tests, but at least one of them is still failing since > then - I’ll go add a comment on the issue. > > -- > Steve > www.lucidworks.com <http://www.lucidworks.com/> > > > On Jun 4, 2018, at 11:16 AM, Erick Erickson > <mailto:erickerick...@gmail.com>> wrote: > > > > Adrien: > > > > "Do you know whether there is something that makes these jobs any > > different?" > > > > Unfortunately no. I'm not really very well versed in the various test > > environments, maybe Uwe or Steve Rowe or Hoss might have some insight? > > > > On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand > <mailto:jpou...@gmail.com>> wrote: > >> Thanks for helping on this front Erick. I noticed a significant decrease in > >> noise since you started badapple-ing bad tests, but I'm observing that our > >> smoke-release builds still keep failing because of Solr tests (10 out of > >> the > >> last 10 builds) in spite of the fact that they disable bad apples: > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/ > >> <https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/> > >> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/ > >> <https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/> > >> > >> Do you know whether there is something that makes these jobs any different? > >> > >> Le mar. 29 mai 2018 à 18:12, Erick Erickson >> <mailto:erickerick...@gmail.com>> a > >> écrit : > >>> > >>> With the long weekend and the fact that the number of non-BadApple > >>> tests is fairly small this week, I'll skip adding more BadApple tests > >>> until next week. > >>> > >>> We're scarily close to the non-BaApple'd tests coming under control. > >>> If we're lucky, we can draw a line in the sand soon then start working > >>> on the backlog. I'll be encouraged if we can start shrinking the > >>> BadApple'd tests. > >>> > >>> - > >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >>> <mailto:dev-unsubscr...@lucene.apache.org> > >>> For additional commands, e-mail: dev-h...@lucene.apache.org > >>> <mailto:dev-h...@lucene.apache.org> > >>> > >> > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > <mailto:dev-unsubscr...@lucene.apache.org> > > For additional commands, e-mail: dev-h...@lucene.apache.org > > <mailto:dev-h...@lucene.apache.org> > > > > > > Begin forwarded message: > > > From: Erick Erickson > <mailto:erickerick...@gmail.com>> > > Subject: Re: Not
