[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680131#comment-17680131
]
Geoff Soutter edited comment on SUREFIRE-2144 at 2/1/23 5:52 AM:
-
OK, so I made a local version of JUnit to explore what a JUnit fix might looks
like. This has ErrorRunningReporter replaced with throw
{{RuntimeException("PREVENTING ...". e)}} in both {{SuiteMethodBuilder}} and
{{{}AllDefaultPossibilitiesBuilder{}}}. When run directly from IDEA, it reports
{{{}"No tests were found"{}}}, which is what we want - no fake tests created.
But when tested with Surefire, slightly different fake test is synthesised. In
this case, we can see the {{initializationFailed}} disappears which makes me
think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:59)
... 9 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection
refused (Connection refused)
{code}
So it seems fixing this problem would require changes in both JUnit and
Surefire. Sigh.
was (Author: gsoutter):
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}, which is what we
want - no fake tests created.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilit
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680109#comment-17680109
]
Geoff Soutter edited comment on SUREFIRE-2144 at 2/1/23 5:51 AM:
-
Did a bit more static analysis and checked the JUnit 4 code from github.
There in RunnerBuilder, it does this
{code}
public Runner safeRunnerForClass(Class testClass) {
try {
Runner runner = runnerForClass(testClass);
if (runner != null) {
configureRunner(runner);
}
return runner;
} catch (Throwable e) {
return new ErrorReportingRunner(testClass, e);
}
}
{code}
ErrorReportingRunner is the thing that synthesizes the fake tests.
{code}
private Description describeCause() {
return Description.createTestDescription(classNames,
"initializationError");
}
{code}
The issue is that SuiteMethodBuilder overrides RunnerBuilder, but doesn't
override safeRunnerForClass, which provides behaviour which is not safe for
Suites (i.e. generated fake fails in place of the actual generated tests).
So from my static analysis, seems this is a JUnit issue rather than Surefire
issue.
I raised https://github.com/junit-team/junit4/issues/1758 for that.
was (Author: gsoutter):
Did a bit more static analysis and checked the JUnit 4 code from github.
There in RunnerBuilder, it does this
{code}
public Runner safeRunnerForClass(Class testClass) {
try {
Runner runner = runnerForClass(testClass);
if (runner != null) {
configureRunner(runner);
}
return runner;
} catch (Throwable e) {
return new ErrorReportingRunner(testClass, e);
}
}
{code}
ErrorReportingRunner is the thing that synthesizes the fake tests.
{code}
private Description describeCause() {
return Description.createTestDescription(classNames,
"initializationError");
}
{code}
The issue is that SuiteMethodBuilder overrides RunnerBuilder, but doesn't
override safeRunnerForClass, which provides behaviour which is not safe for
Suites (i.e. generated fake fails in place of the actual generated tests).
So from my static analysis, seems this is a JUnit issue rather than Surefire
issue.
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.Delegati
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681251#comment-17681251
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/27/23 9:28 AM:
--
I created SUREFIRE-2147 to try and make the equivalent JUnit5 dynamically
created tests reporting work at least as well as JUnit 4 does.
Hopefully this might be an escape hatch for the above ancient JUnit4 issue.
Still I guess I need to test the behaviour when the test method fails before
any tests are created
was (Author: gsoutter):
I created SUREFURE-2147 to try and make the equivalent JUnit5 dynamically
created tests reporting work at least as well as JUnit 4 does.
Hopefully this might be an escape hatch for the above ancient JUnit4 issue.
Still I guess I need to test the behaviour when the test method fails before
any tests are created
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for (Test test: tests) {
> testSuite.addTest(yyy);
> }
> return testSuite;
> }
> {code}
> and then running this using Jenkins CI with Surefire.
> There is a nasty failure pattern which periodically deletes the Jenkins test
> history - it resets the Age of all tests in Jenkins back to 1. The history /
> Age report in Jenkins is key for us as it reveals which commit caused the
> failure. We definitely do not want to lose that information.
> The failure pattern goes like this:
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.064 s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.019
> s <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Connection refused
> (Connection refused)
> ...
> at com.company.package.MySuite.suite(MySuite.java:xx)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at org.apa
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680131#comment-17680131
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 8:04 AM:
--
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}, which is what we
want - no fake tests created.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:59)
... 9 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection
refused (Connection refused)
{code}
So it seems fixing this problem would require changes in both JUnit and
Surefire. Sigh.
was (Author: gsoutter):
OK, so I made a local version of JUnit with the ErrorRunningReporter replaced
with throw {{RuntimeException("PREVENTING ...". e)}} in both
{{SuiteMethodBuilder}} and {{{}AllDefaultPossibilitiesBuilder{}}}. When run
directly from IDEA this reports {{{}"No tests were found"{}}}.
But when tested with Surefire, it still synthesizes a slightly different fake
test. In this case, we can see the {{initializationFailed}} disappears which
makes me think this fake test was being created by Surefire rather than JUnit -
presumably in {{executeTestSet}} as above.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.918 s
<<< FAILURE! - in com.project.package.MySuite
[ERROR] com.project.package.MySuite Time elapsed: 0.912 s <<< ERROR!
java.lang.RuntimeException: PREVENTING AllDefaultPossibilitiesBuilder
ErrorReportingRunner
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.safeRunnerForClass(AllDefaultPossibilitiesBuilder.java:63)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: PREVENTING SuiteMethodBuilder
ErrorReportingRunner
at
org.junit.internal.builders.SuiteMethodBuilder.safeRunnerForClass(SuiteMethodBuilder.java:32)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilde
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680073#comment-17680073
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 1:57 AM:
--
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow allow some Throwables to bubble up.
One way to do that would be to add a simple feature switch (eg
synthesizeTestFailureForInitializationError=false) to control the catch
behaviour.
Another way would be to somehow detect the type of test being run and therefore
have different catch behaviour for dynamically created tests than static ones.
However, I'm not sure if surefire is aware of that, and it would be a much
bigger change.
was (Author: gsoutter):
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow allow some Throwables to bubble up.
Some ideas
* manual: a simple feature switch (eg
synthesizeTestFailureForInitializationError=false) to control the behaviour
* automatic: refactor to make the error handling appropriate for the type of
test being run
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.ap
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680073#comment-17680073
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 1:51 AM:
--
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow allow some Throwables to bubble up.
Some ideas
* manual: a simple feature switch (eg
synthesizeTestFailureForInitializationError=false) to control the behaviour
* automatic: refactor to make the error handling appropriate for the type of
test being run
Does this sound plausible?
was (Author: gsoutter):
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite m
[jira] [Comment Edited] (SUREFIRE-2144) Using JUnit4 TestSuite to create test dynamically, synthetic initializationError failure breaks Jenkins test history
[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680073#comment-17680073
]
Geoff Soutter edited comment on SUREFIRE-2144 at 1/24/23 1:42 AM:
--
I did some basic static analysis of the Surefire code, based on the stack trace
above.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
was (Author: gsoutter):
I did some basic static analysis of the Surefire code.
It seems possible that the cause is in
https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit4/src/main/java/org/apache/maven/surefire/junit4/JUnit4Provider.java
{code}
private void executeTestSet( Class clazz, RunListener reporter, Notifier
notifier, RunModeSetter runMode )
{
long testRunId = classMethodIndexer.indexClass( clazz.getName() );
SimpleReportEntry report =
new SimpleReportEntry( NORMAL_RUN, testRunId, clazz.getName(),
null, null, null, systemProps() );
reporter.testSetStarting( report );
try
{
executeWithRerun( clazz, notifier, runMode );
}
catch ( Throwable e )
{
if ( isFailFast() && e instanceof StoppedByUserException )
{
String reason = e.getClass().getName();
Description skippedTest = createDescription( clazz.getName(),
createIgnored( reason ) );
notifier.fireTestIgnored( skippedTest );
}
else
{
String reportName = report.getName();
String reportSourceName = report.getSourceName();
PojoStackTraceWriter stackWriter = new PojoStackTraceWriter(
reportSourceName, reportName, e );
reporter.testError( withException( NORMAL_RUN, testRunId,
reportSourceName, null, reportName, null,
stackWriter ) );
}
}
finally
{
reporter.testSetCompleted( report );
}
}
{code}
Here it is catching Throwable and then reporting a synthetic error and
completing the test set.
I presume the fix here would be to somehow reduce the scope of the catch to
allow some Throwables to bubble up. This could be a feature switch or perhaps
allow Error to bubble up?
Does this sound plausible?
> Using JUnit4 TestSuite to create test dynamically, synthetic
> initializationError failure breaks Jenkins test history
>
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 2.22.2
>Reporter: Geoff Soutter
>Priority: Major
>
> My team is dynamically creating tests using the JUnit3/4 static suite method
> + TestSuite API
> {code}
> public static junit.framework.Test suite() {
> TestSuite testSuite = new TestSuite(xxx);
> for
