[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
------------------------------------
Summary: Synthetic failures for initialization error breaks Jenkins test
history (was: Creating synthetic failures for initialization error breaks the
Jenkins test history)
> Synthetic failures for initialization error breaks Jenkins test history
> -----------------------------------------------------------------------
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 2.22.2
> Reporter: Geoff Soutter
> Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847
> s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046
> s <<< ERROR!
> {code}
> There is no "initializationError" test in our source code. I presume Surefire
> has created it.
> It seems that the Jenkins JUnit report analysis gets completely confused when
> it does the history analysis in this case. Presumably it tries to compare
> * the "many" passing and failing tests from run N-1 with
> * the single fake test from run N
> After that I presume it decides that those many N-1 tests have been deleted,
> and therefore deletes the history of those tests.
> So it seems that if we value the history analysis, we need to keep the test
> names stable. If so, I suspect the fix is here is that Surefire should never
> create and pretend to run synthetic tests. Rather, the entire run should
> simply fail - I presume throw an exception out of the surefire plugin back to
> maven?
> Hopefully this would then prevent Jenkins from doing the test analysis and it
> will not break the history / Age reporting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
