can we disable this bot already? On Fri, Jun 15, 2018, 7:25 PM Martin Gainty <mgai...@hotmail.com> wrote:
> Erick- > > appears that style mis-application may be categorised as INFO > are mixed in with SEVERE errors > > Would it make sense to filter the errors based on severity ? > > > https://docs.oracle.com/javase/7/docs/api/java/util/logging/Level.html > Level (Java Platform SE 7 ) - Oracle Help Center > <https://docs.oracle.com/javase/7/docs/api/java/util/logging/Level.html> > docs.oracle.com > The Level class defines a set of standard logging levels that can be used > to control logging output. The logging Level objects are ordered and are > specified by ordered integers. > if you know Severity you can triage the SEVERE errors before working down > to INFO errors > > WDYT? > Martin > ______________________________________________ > > > > > ------------------------------ > *From:* Erick Erickson <erickerick...@gmail.com> > *Sent:* Friday, June 15, 2018 1:05 PM > *To:* dev@lucene.apache.org; Mark Miller > *Subject:* Re: Status of solr tests > > Mark (and everyone). > > I'm trying to be somewhat conservative about what I BadApple, at this > point it's only things that have failed every week for the last 4. > Part of that conservatism is to avoid BadApple'ing tests that are > failing and _should_ fail. > > I'm explicitly _not_ delving into any of the causes at all at this > point, it's overwhelming until we reduce the noise as everyone knows. > > So please feel totally free to BadApple anything you know is flakey, > it won't intrude on my turf ;) > > And since I realized I can also report tests that have _not_ failed in > a month that _are_ BadApple'd, we can be a little freer with > BadApple'ing tests since there's a mechanism for un-annotating them > without a lot of tedious effort. > > FWIW. > > On Fri, Jun 15, 2018 at 9:09 AM, Mark Miller <markrmil...@gmail.com> > wrote: > > There is an okay chance I'm going to start making some improvements here > as > > well. I've been working on a very stable set of tests on my starburst > branch > > and will slowly bring in test fixes over time (I've already been making > some > > on that branch for important tests). We should currently be defaulting to > > tests.badapples=false on all solr test runs - it's a joke to try and get > a > > clean run otherwise, and even then somehow 4 or 5 tests that fail > somewhat > > commonly have so far avoided Erick's @BadApple hack and slash. They are > bad > > appled on my dev branch now, but that is currently where any time I have > is > > spent rather than on the main dev branches. > > > > Also, too many flakey tests are introduced because devs are not beasting > or > > beasting well before committing new heavy tests. Perhaps we could add > some > > docs around that. > > > > We have built in beasting support, we need to emphasize that a couple > passes > > on a new test is not sufficient to test it's quality. > > > > - Mark > > > > On Fri, Jun 15, 2018 at 9:46 AM Erick Erickson <erickerick...@gmail.com> > > wrote: > >> > >> (Siiiiggggghhhh) All very true. You're not alone in your frustration. > >> > >> I've been trying to at least BadApple tests that fail consistently, so > >> another option could be to disable BadApple'd tests. My hope has been > >> to get to the point of being able to reliably get clean runs, at least > >> when BadApple'd tests are disabled. > >> > >> From that point I want to draw a line in the sand and immediately > >> address tests that fail that are _not_ BadApple'd. At least then we'll > >> stop getting _worse_. And then we can work on the BadApple'd tests. > >> But as David says, that's not going to be any time soon. It's been a > >> couple of months that I've been trying to just get the tests > >> BadApple'd without even trying to fix any of them. > >> > >> It's particularly pernicious because with all the noise we don't see > >> failures we _should_ see. > >> > >> So I don't have any good short-term answer either. We've built up a > >> very large technical debt in the testing. The first step is to stop > >> adding more debt, which is what I've been working on so far. And > >> that's the easy part.... > >> > >> Siiiiiiiiiiiiiigggggggggghhhhhhhhhh > >> > >> Erick > >> > >> > >> On Fri, Jun 15, 2018 at 5:29 AM, David Smiley <david.w.smi...@gmail.com > > > >> wrote: > >> > (Sigh) I sympathize with your points Simon. I'm +1 to modify the > >> > Lucene-side JIRA QA bot (Yetus) to not execute Solr tests. We can and > >> > are > >> > trying to improve the stability of the Solr tests but even > >> > optimistically > >> > the practical reality is that it won't be good enough anytime soon. > >> > When we > >> > get there, we can reverse this. > >> > > >> > On Fri, Jun 15, 2018 at 3:32 AM Simon Willnauer > >> > <simon.willna...@gmail.com> > >> > wrote: > >> >> > >> >> folks, > >> >> > >> >> I got more active working on IndexWriter and Soft-Deletes etc. in the > >> >> last couple of weeks. It's a blast again and I really enjoy it. The > >> >> one thing that is IMO not acceptable is the status of solr tests. I > >> >> tried so many times to get them passing on several different OSs but > >> >> it seems this is pretty hopepless. It's get's even worse the > >> >> Lucene/Solr QA job literally marks every ticket I attach a patch to > as > >> >> `-1` because of arbitrary solr tests, here is an example: > >> >> > >> >> || Reason || Tests || > >> >> | Failed junit tests | solr.rest.TestManagedResourceStorage | > >> >> | | solr.cloud.autoscaling.SearchRateTriggerIntegrationTest | > >> >> | | solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest | > >> >> | | solr.client.solrj.impl.CloudSolrClientTest | > >> >> | | solr.common.util.TestJsonRecordReader | > >> >> > >> >> Speaking to other committers I hear we should just disable this job. > >> >> Sorry, WTF? > >> >> > >> >> These tests seem to fail all the time, randomly and over and over > >> >> again. This renders the test as entirely useless to me. I even invest > >> >> time (wrong, I invested) looking into it if they are caused by me or > >> >> if I can do something about it. Yet, someone could call me out for > >> >> being responsible for them as a commiter, yes I am hence this email. > I > >> >> don't think I am obliged to fix them. These projects have 50+ > >> >> committers and having a shared codebase doesn't mean everybody has to > >> >> take care of everything. I think we are at the point where if I work > >> >> on Lucene I won't run solr tests at all otherwise there won't be any > >> >> progress. On the other hand solr tests never pass I wonder if the > solr > >> >> code-base gets changes nevertheless? That is again a terrible > >> >> situation. > >> >> > >> >> I spoke to varun and anshum during buzzwords if they can give me > some > >> >> hints what I am doing wrong but it seems like the way it is. I feel > >> >> terrible pushing stuff to our repo still seeing our tests fail. I get > >> >> ~15 build failures from solr tests a day I am not the only one that > >> >> has mail filters to archive them if there isn't a lucene tests in the > >> >> failures. > >> >> > >> >> This is a terrible state folks, how do we fix it? It's the lucene > land > >> >> that get much love on the testing end but that also requires more > work > >> >> on it, I expect solr to do the same. That at the same time requires > >> >> stop pushing new stuff until the situation is under control. The > >> >> effort of marking stuff as bad apples isn't the answer, this requires > >> >> effort from the drivers behind this project. > >> >> > >> >> simon > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> >> > >> > -- > >> > Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker > >> > LinkedIn: http://linkedin.com/in/davidwsmiley | Book: > <http://linkedin.com/in/davidwsmiley> > David Smiley - Lucene/Solr Search Developer / Consultant - D W Smiley LLC > | LinkedIn <http://linkedin.com/in/davidwsmiley> > linkedin.com > View David Smiley’s profile on LinkedIn, the world's largest professional > community. David has 3 jobs listed on their profile. See the complete > profile on LinkedIn and discover David’s connections and jobs at similar > companies. > > > >> > http://www.solrenterprisesearchserver.com > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> > > -- > > - Mark > > about.me/markrmiller > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
