[ https://issues.apache.org/jira/browse/SOLR-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384103#comment-16384103 ]
Erick Erickson commented on SOLR-12028: --------------------------------------- Well, the process is evolving. I saw no use in keeping a separate open JIRA that simply said "this test doesn't succeed very often". I thought that pointing people at the uber-JIRA would provide more context. Whether that's the right call or not is an open question. SOLR-7736: Right, I linked it under "related to" rather than "duplicates", we can add a duplicates link in if you'd like. I put in both JIRAs to allow a complete record, if it's too confusing we can remove, say, the 7736 reference. As for AwaitsFix, the theory (which I may not have adhered to completely, feel free to correct) is that when something is actually known to be broken, i.e. code we can point to and say "this code is broken" doesn't need to be run until, well, the code is fixed. This would include dependent libs, code where the root cause is known but not fixed and the like. AwaitsFix aren't run periodically so disappear from sight. BadApple, OTOH, is "this test doesn't run reliably, we have no clue why". These are also run periodically for the various reports to be able to see. That way they don't get lost. It looked to me like AwaitsFix and BadApple were applied somewhat interchangeably so this is an attempt to bring regularity to the annotations. I reconstructed SOLR-12016 under SOLR-12037 BTW. bq: so now this test is running for all developers As it should IMO unless and until we do one of the following 1> decide it's just a crappy test and remove it 2> identify the root problem is and move it to AwaitsFix and link it to a JIRA with the RCA. 3> fix it and remove the annotation altogether. Note that for <3>, I'm starting by commenting out the annotation and providing the date it was commented out on the theory that if the test starts failing again it would be good information to have. OTOH if we look at this in 6 months and it hasn't failed in that time, the commented-out annotation can be removed. My two main goals are: 1> surface any tests that fail regularly with no RCA for reports etc. 2> get clean runs with BadApple=false to catch regressions I don't particularly care what the mechanics are. You'll see, every Sunday, a report to the dev list for the current BadApple and AwaitsFix annotations, file and method. This should complement your reports (which are way cool BTW). So that's all the theory, I expect a few things to shift around before it all settles out... > BadApple and AwaitsFix annotations usage > ---------------------------------------- > > Key: SOLR-12028 > URL: https://issues.apache.org/jira/browse/SOLR-12028 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests > Reporter: Erick Erickson > Assignee: Erick Erickson > Priority: Major > Attachments: SOLR-12016-buildsystem.patch, > SOLR-12028-sysprops-reproduce.patch, SOLR-12028.patch, SOLR-12028.patch > > > There's a long discussion of this topic at SOLR-12016. Here's a summary: > - BadApple annotations are used for tests that intermittently fail, say < 30% > of the time. Tests that fail more often shold be moved to AwaitsFix. This is, > of course, a judgement call > - AwaitsFix annotations are used for tests that, for some reason, the problem > can't be fixed immediately. Likely reasons are third-party dependencies, > extreme difficulty tracking down, dependency on another JIRA etc. > Jenkins jobs will typically run with BadApple disabled to cut down on noise. > Periodically Jenkins jobs will be run with BadApples enabled so BadApple > tests won't be lost and reports can be generated. Tests that run with > BadApples disabled that fail require _immediate_ attention. > The default for developers is that BadApple is enabled. > If you are working on one of these tests and cannot get the test to fail > locally, it is perfectly acceptable to comment the annotation out. You should > let the dev list know that this is deliberate. > This JIRA is a placeholder for BadApple tests to point to between the times > they're identified as BadApple and they're either fixed or changed to > AwaitsFix or assigned their own JIRA. > I've assigned this to myself to track so I don't lose track of it. No one > person will fix all of these issues, this will be an ongoing technical debt > cleanup effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org