[ https://issues.apache.org/jira/browse/SOLR-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457344#comment-13457344 ]
Yonik Seeley commented on SOLR-3766: ------------------------------------ Some concerns: - These tests should not be excluded from the normal non-jenkins command line "ant test" - Some tests may only be unstable on our freebsd instance (because of undesirable configuration like blackhole). I would not want to see such tests excluded from failing the build on all jenkins/automated tests. This goes back to Mark's request for a dont-run-on-our-freebsd annotation (which makes more sense now that we have other jenkins servers running). - Some tests may only be unstable for certain JVMs There should also be consensus before a test gets thrown into a place where people won't notice (or care about) it's failure. Without that, we'd need to come to consensus on an unambiguous policy about when a test may be marked such that it will no longer fail builds. Maybe an easier solution would still be to send mail on failure, and then people who were not interested could exclude it in a mail filter. > A test group and build plan for known unstable tests > ---------------------------------------------------- > > Key: SOLR-3766 > URL: https://issues.apache.org/jira/browse/SOLR-3766 > Project: Solr > Issue Type: Task > Reporter: Dawid Weiss > Assignee: Dawid Weiss > Priority: Minor > > As discussed on the mailing list and suggested by Hoss: > {code} > 1) slap a special "@UnstableTest" annotation on them > 2) set up a new jenkins job that *only* runs these @UnstableTest jobs > 3) configure this new jenkins job to not send any email > {code} > Point 1 is trivial, 2 I have filed an issue for in the runner -- > https://github.com/carrotsearch/randomizedtesting/issues/122 > that I will take care of shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org