On Thu, 24 Mar 2022 at 01:18, Tibor Digana <[email protected]> wrote:
> Hi Olivier, I have used the Maven branch according to your instructions. I > have used the snapshot version of the plugin and JDK 1.8.0u05. > Both implementations end up with the same build result. The only > difference is the number of IF-Else branches in the implementation. > > I started Maven build twice for your and mine impl with these two configs: > > <argLine>-Xmx1m -XX:MaxPermSize=1m</argLine> > > <argLine>-FFFOOOOO</argLine> > > > 1. The first config would end up with BUILD SUCCESS and it is expected > because JUnit caught the OutOfMemoryError "during testing" and the plugin > has reported it. The OOM did not happen in Surefire with my use case which > we cannot guarantee what the JVM would do in another real scenario. > 2. The second config is the *JVM initialization error* which happens > "before testing" and the build fails with a failure as expected, BUILD > FAILURE. > > Pls comment on these build results. These are expected by us, we can agree > on them, right? > it's what I except yes but for case 2 this is failing directly for me (which is really fine as I definitely prefer this behaviour :) ) [*INFO*] *--- *maven-compiler-plugin:3.10.0:testCompile *(default-testCompile)* @ maven-model* ---* [*INFO*] Changes detected - recompiling the module! [*INFO*] Compiling 38 source files to /Volumes/workspace/dev/sources/maven/maven-core/maven-model/target/test-classes [*INFO*] [*INFO*] *--- *maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test *(default-test)* @ maven-model* ---* [*INFO*] Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider [*INFO*] [*INFO*] ------------------------------------------------------- [*INFO*] T E S T S [*INFO*] ------------------------------------------------------- OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1m; support was removed in 8.0 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main" [*INFO*] [*INFO*] Results: [*INFO*] [*INFO*] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 my config is (but the difference looks to be platform specific) mvn -v *Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)* Maven home: /Users/olamy/softs/maven/apache-maven-3.8.4 Java version: 1.8.0_322, vendor: Temurin, runtime: /Library/Java/JavaVirtualMachines/temurin-8.jdk/Contents/Home/jre Default locale: en_AU, platform encoding: UTF-8 OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac" you definitely need to add IT test to your PR anyway the changes looks good to me. > 1. case > *$ mvn verify -nsu -Dmaven.test.failure.ignore* > > [INFO] --- maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test (default-test) @ > maven-model --- > [INFO] Using auto detected provider > org.apache.maven.surefire.junit4.JUnit4Provider > [INFO] > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > [INFO] Running org.apache.maven.model.ActivationFileTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.583 s - in org.apache.maven.model.ActivationFileTest > [INFO] Running org.apache.maven.model.ActivationOSTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.099 s - in org.apache.maven.model.ActivationOSTest > [INFO] Running org.apache.maven.model.ActivationPropertyTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.136 s - in org.apache.maven.model.ActivationPropertyTest > [INFO] Running org.apache.maven.model.ActivationTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.118 s - in org.apache.maven.model.ActivationTest > [INFO] Running org.apache.maven.model.BuildTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.177 s - in org.apache.maven.model.BuildTest > [INFO] Running org.apache.maven.model.CiManagementTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.22 s - in org.apache.maven.model.CiManagementTest > [INFO] Running org.apache.maven.model.ContributorTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.171 s - in org.apache.maven.model.ContributorTest > [INFO] Running org.apache.maven.model.DependencyManagementTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.211 s - in org.apache.maven.model.DependencyManagementTest > [INFO] Running org.apache.maven.model.DependencyTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.163 s - in org.apache.maven.model.DependencyTest > [INFO] Running org.apache.maven.model.DeploymentRepositoryTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.198 s - in org.apache.maven.model.DeploymentRepositoryTest > [INFO] Running org.apache.maven.model.DeveloperTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.201 s - in org.apache.maven.model.DeveloperTest > [INFO] Running org.apache.maven.model.DistributionManagementTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.193 s - in org.apache.maven.model.DistributionManagementTest > [INFO] Running org.apache.maven.model.ExclusionTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.218 s - in org.apache.maven.model.ExclusionTest > [INFO] Running org.apache.maven.model.ExtensionTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.194 s - in org.apache.maven.model.ExtensionTest > [INFO] Running org.apache.maven.model.IssueManagementTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.206 s - in org.apache.maven.model.IssueManagementTest > [INFO] Running org.apache.maven.model.LicenseTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.216 s - in org.apache.maven.model.LicenseTest > [INFO] Running org.apache.maven.model.MailingListTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.257 s - in org.apache.maven.model.MailingListTest > [INFO] Running org.apache.maven.model.merge.ModelMergerTest > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 6.742 s <<< FAILURE! - in org.apache.maven.model.merge.ModelMergerTest > [ERROR] > org.apache.maven.model.merge.ModelMergerTest.testMergedModelSerialization > Time elapsed: 2.502 s <<< ERROR! > java.lang.OutOfMemoryError: Java heap space > at > org.apache.maven.model.merge.ModelMergerTest.testMergedModelSerialization(ModelMergerTest.java:43) > > [INFO] Running org.apache.maven.model.ModelTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 2.079 s - in org.apache.maven.model.ModelTest > [INFO] Running org.apache.maven.model.NotifierTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.655 s - in org.apache.maven.model.NotifierTest > [INFO] Running org.apache.maven.model.OrganizationTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.512 s - in org.apache.maven.model.OrganizationTest > [INFO] Running org.apache.maven.model.ParentTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.367 s - in org.apache.maven.model.ParentTest > [INFO] Running org.apache.maven.model.PluginConfigurationTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.715 s - in org.apache.maven.model.PluginConfigurationTest > [INFO] Running org.apache.maven.model.PluginContainerTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.316 s - in org.apache.maven.model.PluginContainerTest > [INFO] Running org.apache.maven.model.PluginExecutionTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.451 s - in org.apache.maven.model.PluginExecutionTest > [INFO] Running org.apache.maven.model.PluginManagementTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.972 s - in org.apache.maven.model.PluginManagementTest > [INFO] Running org.apache.maven.model.PluginTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.502 s - in org.apache.maven.model.PluginTest > [INFO] Running org.apache.maven.model.PrerequisitesTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.663 s - in org.apache.maven.model.PrerequisitesTest > [INFO] Running org.apache.maven.model.ProfileTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.202 s - in org.apache.maven.model.ProfileTest > [INFO] Running org.apache.maven.model.RelocationTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.562 s - in org.apache.maven.model.RelocationTest > [INFO] Running org.apache.maven.model.ReportingTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.487 s - in org.apache.maven.model.ReportingTest > [INFO] Running org.apache.maven.model.ReportPluginTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.599 s - in org.apache.maven.model.ReportPluginTest > [INFO] Running org.apache.maven.model.ReportSetTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.287 s - in org.apache.maven.model.ReportSetTest > [INFO] Running org.apache.maven.model.RepositoryPolicyTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 > s - in org.apache.maven.model.RepositoryPolicyTest > [INFO] Running org.apache.maven.model.RepositoryTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.102 s - in org.apache.maven.model.RepositoryTest > [INFO] Running org.apache.maven.model.ResourceTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.85 s - in org.apache.maven.model.ResourceTest > [INFO] Running org.apache.maven.model.ScmTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.237 s - in org.apache.maven.model.ScmTest > [INFO] Running org.apache.maven.model.SiteTest > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 1.074 s - in org.apache.maven.model.SiteTest > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1m; > support was removed in 8.0 > [INFO] > [INFO] Results: > [INFO] > [ERROR] Errors: > [ERROR] ModelMergerTest.testMergedModelSerialization:43 » OutOfMemory > Java heap space > [INFO] > [ERROR] Tests run: 149, Failures: 0, Errors: 1, Skipped: 0 > [INFO] > [ERROR] > > Please refer to > C:\vcs\github\maven-core\maven-model\target\surefire-reports for the > individual test results. > Please refer to dump files (if any exist) [date].dump, > [date]-jvmRun[N].dump and [date].dumpstream. > [INFO] > [INFO] --- maven-jar-plugin:3.2.2:jar (default-jar) @ maven-model --- > [INFO] Building jar: > C:\vcs\github\maven-core\maven-model\target\maven-model-3.8.6-SNAPSHOT.jar > ... > ... > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT: > [INFO] > [INFO] Apache Maven ....................................... SUCCESS [ > 16.395 s] > [INFO] Maven Model ........................................ SUCCESS [ > 57.064 s] > [INFO] Maven Artifact ..................................... SUCCESS [ > 6.102 s] > [INFO] Maven Plugin API ................................... SUCCESS [ > 7.027 s] > [INFO] Maven Builder Support .............................. SUCCESS [ > 2.486 s] > [INFO] Maven Model Builder ................................ SUCCESS [ > 15.773 s] > [INFO] Maven Settings ..................................... SUCCESS [ > 2.945 s] > [INFO] Maven Settings Builder ............................. SUCCESS [ > 5.509 s] > [INFO] Maven Repository Metadata Model .................... SUCCESS [ > 3.903 s] > [INFO] Maven Artifact Resolver Provider ................... SUCCESS [ > 13.878 s] > [INFO] Maven Core ......................................... SUCCESS [01:19 > min] > [INFO] Maven SLF4J Simple Provider ........................ SUCCESS [ > 3.678 s] > [INFO] Maven Embedder ..................................... SUCCESS [ > 15.397 s] > [INFO] Maven Compat ....................................... SUCCESS [ > 53.385 s] > [INFO] Apache Maven Distribution .......................... SUCCESS [ > 27.397 s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 05:11 min > [INFO] Finished at: 2022-03-23T01:15:10+01:00 > [INFO] > ------------------------------------------------------------------------ > > 2. case > *$ mvn verify -nsu -Dmaven.test.failure.ignore* > > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > Unrecognized option: -FFFOOOOO > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > [INFO] > [INFO] Results: > [INFO] > [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT: > [INFO] > [INFO] Apache Maven ....................................... SUCCESS [ > 18.186 s] > [INFO] Maven Model ........................................ FAILURE [ > 12.388 s] > [INFO] Maven Artifact ..................................... SKIPPED > ... ... ... > [INFO] Apache Maven Distribution .......................... SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > > > On Tue, Mar 22, 2022 at 11:44 AM Olivier Lamy <[email protected]> wrote: > >> On Tue, 22 Mar 2022 at 14:40, Tibor Digana <[email protected]> >> wrote: >> >> > Sorry for late reply. >> > >> > I have created a demo project >> > https://github.com/Tibor17/maven.test.failure.ignore/ which simulates >> > OOM. >> > >> > According to the definition of the parameter "maven.failure.test.ignore" >> > the failures (also errors) should be ignored during testing. >> > The word "during" is important because the time when it was a failure >> > before JVM startup it was not a test failure - it is MOJO failure. The >> > author who wrote this config param knew this difference. >> > This perfectly fits JVM illegal arguments like -Xxxx. >> > But I have investigated OOM for the same reason with one JVM. My >> suspicion >> > was not confirmed, see the next... >> > If there are two JVMs, we are breaking the JVM during testing. It is a >> very >> > unlikely situation that the JVM argument would not fail in JVM1 but it >> > would fail in JVM2 later. It may happen only if the user >> > uses @surefire.forkCount@ interpolation but it is again very unlikely >> and >> > the users use it in current working directory configuration. >> > >> > I understand Olivier's PR but I have one objection to the code >> branching in >> > the class *SurefireHelper*. It is based on 3 levels but I have to >> disagree >> > due to there still should be 2 levels as before: >> > 1. ignored failures (my change is only here - yellow - only added >> throwing >> > exception) >> > 2. processes failures >> > >> > and so, I have created a PR >> > https://github.com/apache/maven-surefire/pull/496 >> > Pls have a look into it, thx. >> > This diff looks very simple: >> > >> > if ( reportParameters.isTestFailureIgnore() ) >> > { >> > String errorMessage = createErrorMessage( reportParameters, >> > result, firstForkException ); >> > >> > if ( firstForkException instanceof SurefireBooterForkException ) >> > { >> > throw new MojoExecutionException( errorMessage, >> firstForkException >> > ); >> > >> > } >> > >> > log.error( errorMessage ); >> > } >> > else >> > { >> > throwException( reportParameters, result, firstForkException ); >> > } >> > >> > >> > >> > *$ mvn test* >> > >> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: >> > 2.445 s <<< FAILURE! - in org.apache.maven.OOMTest >> > [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.063 s <<< ERROR! >> > java.lang.OutOfMemoryError: Java heap space >> > at org.apache.maven.OOMTest.test(OOMTest.java:15) >> > [INFO] >> > [INFO] Results: >> > [INFO] >> > [ERROR] Errors: >> > [ERROR] OOMTest.test:15 » OutOfMemory Java heap space >> > [INFO] >> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 >> > [INFO] >> > [INFO] >> > ------------------------------------------------------------------------ >> > [INFO] BUILD FAILURE >> > [INFO] >> > ------------------------------------------------------------------------ >> > >> > *$ mvn test -Dmaven.test.failure.ignore* >> > >> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: >> > 2.472 s <<< FAILURE! - in org.apache.maven.OOMTest >> > [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.262 s <<< ERROR! >> > java.lang.OutOfMemoryError: Java heap space >> > at org.apache.maven.OOMTest.test(OOMTest.java:15) >> > [INFO] >> > [INFO] Results: >> > [INFO] >> > [ERROR] Errors: >> > [ERROR] OOMTest.test:15 » OutOfMemory Java heap space >> > [INFO] >> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 >> > [INFO] >> > [ERROR] >> > >> > Please refer to >> > C:\vcs\github\maven.test.failure.ignore\target\surefire-reports for the >> > individual test results. >> > Please refer to dump files (if any exist) [date].dump, >> > [date]-jvmRun[N].dump and [date].dumpstream. >> > [INFO] >> > ------------------------------------------------------------------------ >> > [INFO] BUILD SUCCESS >> > [INFO] >> > ------------------------------------------------------------------------ >> > >> >> Please use a project with multiple modules, not a single one where you can >> see the line with OOME. >> Any project with a huge number of modules will hide this and nobody will >> ever notice it. >> >> >> git clone -b first-module-fail https://github.com/apache/maven >> maven-core >> >> cd maven-core >> >> >> [INFO] --- maven-assembly-plugin:3.3.0:single (create-distro-packages) @ >> apache-maven --- >> [INFO] Reading assembly descriptor: src/main/assembly/bin.xml >> [INFO] Building zip: >> >> /Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.zip >> [INFO] Building tar: >> >> /Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.tar.gz >> [INFO] >> [INFO] --- maven-checkstyle-plugin:3.0.0:check (checkstyle-check) @ >> apache-maven --- >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT: >> [INFO] >> [INFO] Apache Maven ....................................... SUCCESS [ >> 3.220 s] >> [INFO] Maven Model ........................................ SUCCESS [ >> 4.734 s] >> [INFO] Maven Artifact ..................................... SUCCESS [ >> 3.142 s] >> [INFO] Maven Plugin API ................................... SUCCESS [ >> 1.763 s] >> [INFO] Maven Builder Support .............................. SUCCESS [ >> 1.288 s] >> [INFO] Maven Model Builder ................................ SUCCESS [ >> 5.129 s] >> [INFO] Maven Settings ..................................... SUCCESS [ >> 0.694 s] >> [INFO] Maven Settings Builder ............................. SUCCESS [ >> 1.730 s] >> [INFO] Maven Repository Metadata Model .................... SUCCESS [ >> 1.470 s] >> [INFO] Maven Artifact Resolver Provider ................... SUCCESS [ >> 4.255 s] >> [INFO] Maven Core ......................................... SUCCESS [ >> 19.075 s] >> [INFO] Maven SLF4J Simple Provider ........................ SUCCESS [ >> 0.972 s] >> [INFO] Maven Embedder ..................................... SUCCESS [ >> 4.293 s] >> [INFO] Maven Compat ....................................... SUCCESS [ >> 10.350 s] >> [INFO] Apache Maven Distribution .......................... SUCCESS [ >> 4.898 s] >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD SUCCESS >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 01:07 min >> [INFO] Finished at: 2022-03-22T20:19:24+10:00 >> [INFO] >> ------------------------------------------------------------------------ >> >> >> >> >> even CI is green >> >> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven/job/first-module-fail/ >> so all looks good to merge a PR... >> but hey in fact a module has not run any tests... >> Do you really think it's right? (yes this problem exists for a very very >> long time but we can choose to fix problems even after a long period) >> Does your PR fix that? >> >> >> >> > >> > >> > >> > On Sat, Mar 19, 2022 at 12:17 PM Slawomir Jaranowski < >> > [email protected]> >> > wrote: >> > >> > > Hi, >> > > >> > > I start to feel that we first try to define technical details but we >> > don't >> > > know for what. >> > > Each business requirement should be resolved as simply as possible >> from a >> > > technical perspective. >> > > In other ways we will start to build complicated and many features >> which >> > > probably were not needed at all. >> > > >> > > Please first define business requirements with examples, next we can >> find >> > > technical ways to cover it. >> > > >> > > Example scenario: >> > > >> > > Businesses requirement: >> > > We want to collect history of test result in order to detect flaky >> > tests, >> > > analize tests execution time, etc >> > > >> > > Given: >> > > - build is done by CI system >> > > - there is system which collect reports for executed tests and build >> > > history of it >> > > - there are broken tests in build - we know them and accept (eg, on >> some >> > > os) >> > > >> > > When: >> > > - build finished with success on CI >> > > >> > > Then: >> > > - all tests result are reported >> > > - failed tests are reported >> > > >> > > >> > > >> > > >> > > sob., 19 mar 2022 o 06:27 Christoph Läubrich <[email protected]> >> > > napisał(a): >> > > >> > > > Hi Tibor, >> > > > >> > > > just to make it clear, I don't talkin about log-levels here but how >> to >> > > > intepret/handle a failure/failing/crashing test. >> > > > >> > > > >> > > > Am 18.03.22 um 21:54 schrieb Tibor Digaňa: >> > > > > Christoph, just let me briefly explain the log level hierarchy. >> > > > > If you select WARN log level, then ERROR can be printed too. >> > > > > Similar with INFO, means that WARN and ERROR would be printed as >> > well. >> > > > > >> > > > > The real decision making in the plugin is a bit more complicated, >> see >> > > the >> > > > > pull request https://github.com/apache/maven-surefire/pull/478 >> > > > > After I have replied to this email, I would verify the behavior of >> > the >> > > > > plugin simulating OOM. >> > > > > >> > > > > >> > > > > T >> > > > > >> > > > > >> > > > > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich < >> > > [email protected]> >> > > > > wrote: >> > > > > >> > > > >> No I think more about >> > > > >> >> > > > >> if (severity == "WARN") { >> > > > >> log.warn("There is something wrong"); >> > > > >> } else if (severity == "ERROR") { >> > > > >> throw new MojoFailureException("...") { >> > > > >> } else { >> > > > >> throw new MojoExecutionException("...") { >> > > > >> } >> > > > >> >> > > > >> That way the plugin can handle any error/failure/... in a way >> that >> > the >> > > > >> user can "downgrade" a certain problem if desired. >> > > > >> >> > > > >> >> > > > >> Am 18.03.22 um 00:06 schrieb Tibor Digana: >> > > > >>> @Christoph >> > > > >>> FATAL , WARN , ERROR >> > > > >>> They are log levels? >> > > > >>> The plugin does not control the log level after caught >> exception in >> > > > Maven >> > > > >>> core. The Maven Core does! >> > > > >>> I think it's time to make a demo. >> > > > >>> >> > > > >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich < >> > > > [email protected]> >> > > > >>> wrote: >> > > > >>> >> > > > >>>> Hi Tibor, >> > > > >>>> >> > > > >>>> it shouldn't be to hard to guess another property like >> > > > >>>> >> > > > >>>> maven.test.jvmcrash=FATAL >> > > > >>>> maven.test.failure=WARN >> > > > >>>> maven.test.error=ERROR >> > > > >>>> >> > > > >>>> :-) >> > > > >>>> >> > > > >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana: >> > > > >>>>> Hi Christoph, >> > > > >>>>> >> > > > >>>>> Such a granularity with error/failure might be also an >> additional >> > > > >>>>> requirement but still you miss the third option to JVM error >> > which >> > > is >> > > > >>>>> different from test error/failure - they don;t have the same >> > > meaning. >> > > > >>>>> >> > > > >>>>> T >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich < >> > > > >> [email protected] >> > > > >>>>> >> > > > >>>>> wrote: >> > > > >>>>> >> > > > >>>>>> Just wondering but maybe it would be better to configure the >> > > > severity >> > > > >>>>>> instead of a true/false, something like: >> > > > >>>>>> >> > > > >>>>>> maven.test.failure=WARN >> > > > >>>>>> maven.test.error=ERROR >> > > > >>>>>> >> > > > >>>>>> would only warn about failing tests but thrw exception if >> > starting >> > > > the >> > > > >>>>>> test fails? >> > > > >>>>>> >> > > > >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana: >> > > > >>>>>>> Romain, it is not a bug. >> > > > >>>>>>> Don't consider this as a bug. It was a feature request for >> > change >> > > > by >> > > > >>>>>>> Olivier, and not a bug. >> > > > >>>>>>> I closed both issues years ago but not because of ignorance >> but >> > > > >> because >> > > > >>>>>> the >> > > > >>>>>>> appearance of the exceptional behavior is a wrong compromise >> > and >> > > > >> which >> > > > >>>>>> does >> > > > >>>>>>> not help anyone and even it makes the situation worse >> because >> > > > >> typically >> > > > >>>>>>> other group of users would fire a new Jira tickets. You >> would >> > not >> > > > be >> > > > >>>> able >> > > > >>>>>>> to satisfy two contradictory parties which have completely >> > > > different >> > > > >>>>>>> opinions, and so we use to introduce new params and this >> way we >> > > > >> satisfy >> > > > >>>>>>> both parties, they may combine the params as they wish, and >> > > > everybody >> > > > >>>>>> would >> > > > >>>>>>> be happy and nobody would claim. Many technical solutions >> might >> > > > >> exist, >> > > > >>>>>> e.g. >> > > > >>>>>>> param=boolean|string or new param=boolean, whatever. >> > > > >>>>>>> >> > > > >>>>>>> The truth is that this problem with (java --add-reads ...) >> > > happened >> > > > >> in >> > > > >>>>>> our >> > > > >>>>>>> ASF environment on Java 8 which in good configuration would >> not >> > > > >> happen >> > > > >>>>>> and >> > > > >>>>>>> should not. >> > > > >>>>>>> It is not right way that we abuse >> "maven.test.failure.ignore" >> > > which >> > > > >>>> would >> > > > >>>>>>> tell us "Hey, you have illegal configuration in your build >> > system >> > > > and >> > > > >>>> it >> > > > >>>>>>> would not work by JDK design", it is not the goal of the >> plugin >> > > to >> > > > >> tell >> > > > >>>>>> you >> > > > >>>>>>> that you have configured the build wrong because it won't >> and >> > the >> > > > >> same >> > > > >>>>>>> configuration of the plugin (not the build) says "ignore >> the >> > > > error". >> > > > >>>>>>> Yesterday I discussed this problem with Herve and we >> > > independently >> > > > >>>>>> observed >> > > > >>>>>>> equal opinions and that's not everything because we >> understood >> > > that >> > > > >> the >> > > > >>>>>>> purpose of the config parameter is to not throw MOJO >> exception >> > > > which >> > > > >> is >> > > > >>>>>>> right, but the exceptional behavior should have an exact new >> > > param >> > > > >> for >> > > > >>>>>> the >> > > > >>>>>>> exact use case. >> > > > >>>>>>> SPI for this parameter is too much because no user would >> > > implement >> > > > it >> > > > >>>>>> for a >> > > > >>>>>>> trivial parameter;; the SPI is used to be implemented by >> > > frameworks >> > > > >> or >> > > > >>>>>>> systems or application servers but this is not our case. >> > > > >>>>>>> >> > > > >>>>>>> >> > > > >>>>>>> >> > > > >>>>>>> >> > > > >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau < >> > > > >>>>>> [email protected]> >> > > > >>>>>>> wrote: >> > > > >>>>>>> >> > > > >>>>>>>> +1 >> > > > >>>>>>>> if it is to investigate a CI issue, it is generally easy to >> > add >> > > > >> debug >> > > > >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest >> > for >> > > > the >> > > > >>>>>> plugin >> > > > >>>>>>>> to me. >> > > > >>>>>>>> >> > > > >>>>>>>> Romain Manni-Bucau >> > > > >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> > > > >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >> > > > >>>>>>>> <http://rmannibucau.wordpress.com> | Github < >> > > > >>>>>>>> https://github.com/rmannibucau> | >> > > > >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book >> > > > >>>>>>>> < >> > > > >>>>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >> >> > > > >> > > >> > >> https://www.packtpub.com/application-development/java-ee-8-high-performance >> > > > >>>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet < >> > > [email protected]> >> > > > a >> > > > >>>>>> écrit >> > > > >>>>>>>> : >> > > > >>>>>>>> >> > > > >>>>>>>>> If that's not currently possible, maybe a SPI should be >> > > provided >> > > > so >> > > > >>>>>> that >> > > > >>>>>>>>> people can use plug in extensions to analyze the test >> result >> > > and >> > > > >>>>>> override >> > > > >>>>>>>>> it if necessary (transforming an error into a warning, >> > storing >> > > > >>>> results >> > > > >>>>>>>> in a >> > > > >>>>>>>>> way which is easier to use by other tools later...) ? >> > > > >>>>>>>>> >> > > > >>>>>>>>> Guillaume >> > > > >>>>>>>>> >> > > > >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich < >> > > > >>>> [email protected]> >> > > > >>>>>> a >> > > > >>>>>>>>> écrit : >> > > > >>>>>>>>> >> > > > >>>>>>>>>> I also agree that the test at least should run, we use >> this >> > > > >> property >> > > > >>>>>> to >> > > > >>>>>>>>>> run the test and produce result and later on have a >> > buildstep >> > > > that >> > > > >>>>>>>>>> analyze the results (and probably fail the build job). >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> As it is not recommend, I wonder what is the recommended >> way >> > > to >> > > > >>>>>> archive >> > > > >>>>>>>>>> something similar? >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy: >> > > > >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana < >> > > > >> [email protected] >> > > > >>>>> >> > > > >>>>>>>>>> wrote: >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>>> In case of the user property >> *maven.test.failure.ignore* >> > the >> > > > >> MOJO >> > > > >>>>>>>> must >> > > > >>>>>>>>>> not >> > > > >>>>>>>>>>>> throw any exception which is interpreted by the Maven >> Core >> > > as >> > > > >>>> BUILD >> > > > >>>>>>>>>>>> SUCCESS. >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> This is a very simple reduction of the problem >> description. >> > > > >>>>>>>>>>> The documentation here >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>> >> > > > >>>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >> >> > > > >> > > >> > >> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore >> > > > >>>>>>>>>>> says >> > > > >>>>>>>>>>> "Set this to "true" to ignore a failure during testing. >> Its >> > > use >> > > > >> is >> > > > >>>>>>>> NOT >> > > > >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion" >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> Personally, I understand this to ignore failures in >> junit >> > > > results >> > > > >>>> BUT >> > > > >>>>>>>>> at >> > > > >>>>>>>>>>> least I want the tests to run. >> > > > >>>>>>>>>>> I guess this is how our users use this feature (feature >> we >> > do >> > > > not >> > > > >>>>>>>>>> recommend >> > > > >>>>>>>>>>> by the way...) >> > > > >>>>>>>>>>> And this is perfectly explained here >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>> >> > > > >>>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >> >> > > > >> > > >> > >> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077 >> > > > >>>>>>>>>>> there is a difference between ignoring some junit >> failures >> > > and >> > > > >>>>>>>>> ignoring a >> > > > >>>>>>>>>>> configuration error because some jvm args has been >> > > > misconfigured >> > > > >>>> for >> > > > >>>>>>>>> many >> > > > >>>>>>>>>>> reasons and surefire cannot start. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any >> > > exception >> > > > ") >> > > > >>>> we >> > > > >>>>>>>>>> should >> > > > >>>>>>>>>>> ignore every surefire configuration error and keep >> running >> > > the >> > > > >>>> build >> > > > >>>>>>>>>> until >> > > > >>>>>>>>>>> the end and says BUILD SUCCESS >> > > > >>>>>>>>>>> what about >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber >> > > > >>>>>>>>>>> -Dmaven.test.failure.ignore=true >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> we should not throw any exceptions as you said below and >> > keep >> > > > >>>> saying >> > > > >>>>>>>>>> BUILD >> > > > >>>>>>>>>>> SUCCESS? >> > > > >>>>>>>>>>> argLine is a configuration element as any others so it >> > should >> > > > >> fail >> > > > >>>>>>>>>> directly >> > > > >>>>>>>>>>> and not be ignored especially when the surefire plugin >> > cannot >> > > > >> run. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>>> We have received an internal requirement to change the >> > > > behavior >> > > > >> of >> > > > >>>>>>>> the >> > > > >>>>>>>>>> user >> > > > >>>>>>>>>>>> property *maven.test.failure.ignore* so that the >> behavior >> > > will >> > > > >>>> have >> > > > >>>>>>>>> one >> > > > >>>>>>>>>>>> exception. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> Suppose that you have JDK 1.8 but you use: >> > > > >>>>>>>>>>>> /jre/java --add-reads ... >> > > > >>>>>>>>>>>> The outcome is JVM exit with an error message. >> > > > >>>>>>>>>>>> I agree with Herve who also says that >> > > > >> *maven.test.failure.ignore* >> > > > >>>>>>>>>> should >> > > > >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a >> > > typical >> > > > >> JVM >> > > > >>>>>>>>>>>> segmentation fault or another JVM crash where we >> cannot do >> > > > >>>> anything >> > > > >>>>>>>>>> about >> > > > >>>>>>>>>>>> it, and where the entire build would crash in the >> command >> > > line >> > > > >>>> which >> > > > >>>>>>>>>>>> of course means that the build cannot normally be >> > > interpreted >> > > > as >> > > > >>>>>>>> BUILD >> > > > >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures >> > > related >> > > > >> to >> > > > >>>>>>>> the >> > > > >>>>>>>>>> test >> > > > >>>>>>>>>>>> purposes. >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> On the other hand, Olivier has reopened the issues >> > > > SUREFIRE-1426 >> > > > >>>> and >> > > > >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the >> > feature >> > > is >> > > > >>>>>>>>> expected. >> > > > >>>>>>>>>>>> This is exactly the reason why I closed these tickets >> > > several >> > > > >>>> years >> > > > >>>>>>>>> ago >> > > > >>>>>>>>>> and >> > > > >>>>>>>>>>>> my proposal was to not to have any exceptions in the >> > feature >> > > > >>>>>>>> behavior >> > > > >>>>>>>>>> and >> > > > >>>>>>>>>>>> the proposal was to introduce a new user property for >> > exact >> > > > use >> > > > >>>>>>>> cases. >> > > > >>>>>>>>>>>> The next idea, which comes from two developers, would >> > > provide >> > > > us >> > > > >>>>>>>> with >> > > > >>>>>>>>>> the >> > > > >>>>>>>>>>>> same non exceptional and exact behavior of the user >> > property >> > > > >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide >> us >> > > with >> > > > >> new >> > > > >>>>>>>> user >> > > > >>>>>>>>>>>> property in the case with fine grade control of the >> build >> > > > >> errors, >> > > > >>>>>>>> e.g. >> > > > >>>>>>>>>>>> *maven.test.jvm.error.ignore*. >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> with a default of? >> > > > >>>>>>>>>>> honestly I just see this new parameter as introducing >> more >> > > > >>>> complexity >> > > > >>>>>>>>> in >> > > > >>>>>>>>>> an >> > > > >>>>>>>>>>> already very complex code and not worth it. >> > > > >>>>>>>>>>> again read both issue's comments and my comments. >> > > > >>>>>>>>>>> Please try to have a user POV and think about making our >> > > users' >> > > > >>>>>>>>>> experience >> > > > >>>>>>>>>>> more simple. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> This should be very simple: >> > > > >>>>>>>>>>> If surefire forked jvm cannot start it's build error and >> > > cannot >> > > > >> run >> > > > >>>>>>>> any >> > > > >>>>>>>>>>> tests, it's a problem users want to know immediately >> > because >> > > it >> > > > >> can >> > > > >>>>>>>> be >> > > > >>>>>>>>>> for >> > > > >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on >> the >> > > > system >> > > > >>>>>>>> etc... >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> AND AGAIN it is very different from ignoring junit >> result >> > > > >> failures. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> Try to look at a user point of view and think about >> what is >> > > the >> > > > >> use >> > > > >>>>>>>>> case >> > > > >>>>>>>>>> of >> > > > >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess >> 99% of >> > > the >> > > > >>>> time, >> > > > >>>>>>>>>> users >> > > > >>>>>>>>>>> wants to run all their tests (even on a CI with >> different >> > > > matrix) >> > > > >>>> so >> > > > >>>>>>>>> they >> > > > >>>>>>>>>>> can collect all the tests results which has runned to >> see >> > if >> > > > >> there >> > > > >>>> is >> > > > >>>>>>>>> any >> > > > >>>>>>>>>>> issue for some combination of their matrix and >> eventually >> > > turn >> > > > >> the >> > > > >>>>>>>>> result >> > > > >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins >> and >> > > was >> > > > >>>> even >> > > > >>>>>>>> a >> > > > >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin) >> > > > >>>>>>>>>>> BUT if for any reasons one of the module do not start >> his >> > > tests >> > > > >>>>>>>> because >> > > > >>>>>>>>>> the >> > > > >>>>>>>>>>> jvm cannot be start the users will not see that and >> will be >> > > > >> totally >> > > > >>>>>>>>> blind >> > > > >>>>>>>>>>> until maybe someone look inside a very very large log >> file >> > > > (which >> > > > >>>>>>>> means >> > > > >>>>>>>>>>> probably never) >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> Long story short as my experience as a user facing this >> > > > >>>> problem/bug: >> > > > >>>>>>>>>>> I had the case on a very large multi modules (~250 >> modules) >> > > > build >> > > > >>>> of >> > > > >>>>>>>> a >> > > > >>>>>>>>>> very >> > > > >>>>>>>>>>> used open source project. >> > > > >>>>>>>>>>> I was using this maven.test.failure.ignore property but >> one >> > > of >> > > > >> the >> > > > >>>>>>>>>> modules >> > > > >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the >> CI. >> > > > >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different >> > maven >> > > > run >> > > > >>>>>>>> which >> > > > >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M >> > > > >>>>>>>>>>> So because of this property the build was not failing >> and >> > > kept >> > > > >>>> saying >> > > > >>>>>>>>>> BUILD >> > > > >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one >> > module >> > > > >> with >> > > > >>>>>>>> jdk >> > > > >>>>>>>>>> 17... >> > > > >>>>>>>>>>> And frankly you do not dig into a log file of 32M after >> > each >> > > > run >> > > > >>>>>>>>>> especially >> > > > >>>>>>>>>>> when it says success... >> > > > >>>>>>>>>>> 3 days after the first release claiming supporting jdk >> 17 >> > we >> > > > >>>>>>>> received a >> > > > >>>>>>>>>> bug >> > > > >>>>>>>>>>> report about a something not working with jdk17.... >> > > > >>>>>>>>>>> and guess what? Where was this feature supposed to be >> > tested? >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> so I frankly believe we do not need a complex new >> property, >> > > in >> > > > >> this >> > > > >>>>>>>>> case >> > > > >>>>>>>>>>> this should fail directly because this will improve user >> > > > >>>> experience. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> I would like to open the discussion on this topic. >> You're >> > > > >> welcome! >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> Cheers >> > > > >>>>>>>>>>>> Tibor >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > > >>>> >> > > --------------------------------------------------------------------- >> > > > >>>>>>>>>> To unsubscribe, e-mail: [email protected] >> > > > >>>>>>>>>> For additional commands, e-mail: >> [email protected] >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > > >>>>>>>>> >> > > > >>>>>>>>> -- >> > > > >>>>>>>>> ------------------------ >> > > > >>>>>>>>> Guillaume Nodet >> > > > >>>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>> >> > > > >>>>>> >> > > > >>>>>> >> > > > >> --------------------------------------------------------------------- >> > > > >>>>>> To unsubscribe, e-mail: [email protected] >> > > > >>>>>> For additional commands, e-mail: [email protected] >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>>> >> > > > >>>> >> > > > >>>> >> > > --------------------------------------------------------------------- >> > > > >>>> To unsubscribe, e-mail: [email protected] >> > > > >>>> For additional commands, e-mail: [email protected] >> > > > >>>> >> > > > >>>> >> > > > >>> >> > > > >> >> > > > >> >> > --------------------------------------------------------------------- >> > > > >> To unsubscribe, e-mail: [email protected] >> > > > >> For additional commands, e-mail: [email protected] >> > > > >> >> > > > >> >> > > > > >> > > > >> > > > >> --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: [email protected] >> > > > For additional commands, e-mail: [email protected] >> > > > >> > > > >> > > >> > > -- >> > > Sławomir Jaranowski >> > > >> > >> >
