[JIRA] [junit] (JENKINS-14089) JUnit report - Quarantine intermittent tests

Tue, 19 Aug 2014 12:48:19 -0700

 
samuel nobs edited a comment on New Feature JENKINS-14089

here's what i posted on the jenkins developers list a while ago. if someone could give me a few hints i might be able to move this forward:

hello everyone

i have created a plugin that adds the ability to quarantine a test case that fails intermittently:

https://github.com/samsta/quarantine

we are using it on our jenkins instance, but it has one major area of concern. because the build status can't be made better, but only worse, i can't just run another step after the test result archiver, so i had to piggy-back functionality on a sub-class of JUnitResultArchiver:

https://github.com/samsta/quarantine/blob/master/src/main/java/org/jenkinsci/plugins/quarantine/QuarantinableJUnitResultArchiver.java

personally, i do not like this approach because of the following issues:

  • it is brittle
  • it is confusing and error prone in the job configuration
  • it is too tightly coupled to the junit result format

i would really like to resolve this before releasing this plugin into the wider world as i am not really willing to provide support, knowing of these architectural deficiencies.

can anyone suggest a way forward, please? all ideas i have so far require changes in the core:

  • add an extension point that allows to assess test results after the archiver has run
  • this would mean that the archiver is no longer making the decision about the build status
  • not really backwards compatible
  • drop the requirement that builds can only be made worse
  • yuck! potentially opens a can of worms
  • allow to modify a test result from within a TestAction or TestDataPublisher, either using the skipped state or by adding a new ignored/quarantined state
  • test result has been immutable so far
  • a fair bit of work in TestResult and friends, but these could do with some work because the relation between the hudson.tasks.test.TestResult and hudson.tasks.junit.TestResult is a bit confusing

cheers,
sam
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to