[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325240#comment-16325240
 ] 

Hudson commented on SUREFIRE-1264:
--

ABORTED: Integrated in Jenkins build maven-surefire-dev #103 (See 
[https://builds.apache.org/job/maven-surefire-dev/103/])
[SUREFIRE-1264] Some tests can be lost when running in parallel with (tibor17: 
[http://git-wip-us.apache.org/repos/asf/?p=maven-surefire.git=commit=c8724dd2f992625c355e4817a6b7995d3e88fdb4])
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/ATest.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/BaseTest.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/TestSet.java
* (edit) 
surefire-integration-tests/src/test/resources/surefire-621-testCounting-junit3-in-parallel/pom.xml
* (edit) 
surefire-api/src/main/java/org/apache/maven/surefire/util/internal/TestClassMethodNameUtils.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/CTest.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-621-testCounting-junit3-in-parallel/src/it/java/mho/JUnit4AdapterSuiteTest.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/BTest.java
* (edit) 
surefire-providers/surefire-junit47/src/test/java/org/apache/maven/surefire/junitcore/TestMethodTest.java
* (edit) 
surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/junit4/MockReporter.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/DTest.java
* (edit) 
surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/common/junit4/JUnit4RunListener.java
* (add) surefire-integration-tests/src/test/resources/surefire-1264/pom.xml
* (add) 
surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/jiras/Surefire1264IT.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/ConcurrentRunListener.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreRunListener.java
* (edit) 
surefire-providers/surefire-junit47/src/test/java/org/apache/maven/surefire/junitcore/JUnitCoreRunListenerTest.java
* (edit) 
surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/jiras/Surefire621TestCountingJunit3InParallelIT.java


> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193571#comment-16193571
 ] 

Christian Beikov commented on SUREFIRE-1264:


Here it is: SUREFIRE-1430

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193503#comment-16193503
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~christian.beikov]
I have reproduced the issue with minimum test on real box.
Can you please report a Jira issue related to platform Linux/Ubuntu with 
external link pointing to github?
Current issue is about different root cause.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193415#comment-16193415
 ] 

Christian Beikov commented on SUREFIRE-1264:


Here is a minimal test: https://github.com/beikov/surefire-test

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193407#comment-16193407
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
I am able to reproduce this issue on real box, Ubuntu 17.
I have investigated this issue a little bit.
It really looks like Surefire issue, but I cannot confirm unless isolated test 
reproduces the issue with minimum dependencies. The fact is that there is JUnit 
suite {{CalciteSuite.java}} has {{QuidemTest.java}} which is {{@Parameterized}} 
JUnit test and the filter is specified 
{{org/apache/calcite/test/CalciteSuite.java}}.
It really does not matter how many Threads you allocate.
Would you isolate it in a separate project in your account on GitHub? I want to 
know if the single threaded Jdbc is the issue or dependencies. Let's avoid them.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193052#comment-16193052
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
I guess you are using this configuration:
{quote}
1
true
both
once
{quote}
Try to change it to this:
{quote}
2
4
false
classesAndMethods
once
{quote}
I changed 4 parameters. Two classes run in parallel and 4 methods.
Let me know when Travis CI has finished.
I need to know if I should investigate parallel computer issue.
Thx.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192759#comment-16192759
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
Maybe we have talked about it but I would ask again. 
Do you have a Thread Dump when Surefire 2.18 hanged in project 
{{calcite-core}}? 
Can you attach it?
Please use version 2.18 and not 2.18.1.
The old version definitely did not have new threading mechanism when Maven sent 
tests to run to Surefire process via native process pipes.
It looks to me like the test class name was swallowed somewhere.
Are you open for building some Surefire branch by yourselves and perform some 
experiments on your side?
Perhaps it would be nice dumping this cross-process communication when 
debugging Maven build like this {{mvn -X test}} or {{mvn --debug test}}.
Thx.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180668#comment-16180668
 ] 

Christian Beikov commented on SUREFIRE-1264:


I just tried on a VMware virtualized RHEL and had the same problem. Starting to 
think this might be a general problem on Linux.
I don't have access to a Linux installed on bare-metal, but who would run a 
build on a bare-metal installed OS anyway? Isn't everything virtualized?
The surefire-test project perfectly reproduces the issue, so you should be able 
to dig into the real problem when you have time.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180639#comment-16180639
 ] 

Tibor Digana commented on SUREFIRE-1264:


I did not have time yet. Please check it out on real Linux first.

On Tue, Sep 26, 2017 at 11:08 AM, Christian Beikov (JIRA) 



> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180512#comment-16180512
 ] 

Christian Beikov commented on SUREFIRE-1264:


I haven't tried on a non-virtualized Linux machine myself yet and I don't know 
if the problems might be because of virtualization, but I just tried to run on 
Travis CI with sudo enabled i.e. on a directly virtualized Linux without Docker 
and it is stuck again: 
https://travis-ci.org/beikov/surefire-test/builds/27987
I don't have a non-virtualized Linux handy, but I am not sure why that would 
matter. Have you been able to test on Fedora or Ubuntu yet?

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176810#comment-16176810
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
I will have time on Sunday evening but meanwhile I found articles related to 
Docker, MMU and minimum installation.
Has this build ever run on Ubuntu or another non-virtualized Linux without 
Docker?
https://www.sitepoint.com/docker-containers-software-delivery/
https://blog.codeship.com/docker-for-windows-linux-and-mac/




> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176688#comment-16176688
 ] 

Tibor Digana commented on SUREFIRE-1264:


I do not have docker but I will check it our on Fedora and Ubuntu.
Did you have the same problem on ordinal Linux not running under Docker?
Does it have to do something with MMU in CPU due to Linux Virtualization of
Kernel?

On Fri, Sep 22, 2017 at 10:18 AM, Christian Beikov (JIRA) 



> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176101#comment-16176101
 ] 

Christian Beikov commented on SUREFIRE-1264:


The problem definitely is the fact that I am executing a Parameterized test 
with parallelism enabled in surefire. I disabled all other tests of the suite 
already, so there shouldn't be any other side effects. The calcite-linq4j has 
only normal tests, no parameterized tests. It's not a disk space problem, the 
build runs through without parallelism as you can see here: 
https://travis-ci.org/beikov/calcite/builds/278513374

The problem appears only in this combination AFAICT: Docker with 2 CPUs, 
Parameterized test and surefire with parallelism enabled
Here is even a minimal test case(https://github.com/beikov/surefire-test) that 
shows the problem: https://travis-ci.org/beikov/surefire-test/builds/278521316

I suspect this might be a general Linux problem as the build works fine on 
Windows. Hope this helps you to find the issue.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175279#comment-16175279
 ] 

Tibor Digana commented on SUREFIRE-1264:


We have experiences on FreeBSD when forked JVM hanged but did not start any 
test because there was no free space on the disk. Could it be here on Docker?
Would you run the build on your PC with Linux but without Docker?

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175227#comment-16175227
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
Why the project {{calcite-linq4j}} did not hang and {{calcite-core}} did?
Can you make several experiments and exclude some influences, like e.g. 
parallelism in Surefire, and then tests so use empty suite {{CalciteSuite}}, 
then write Test class with one but empty test method and use only that one in 
{{CalciteSuite}}, and then always divide the range of tests in {{CalciteSuite}} 
by two and logarithmically iterate to few tests which are related to our issue.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174433#comment-16174433
 ] 

Christian Beikov commented on SUREFIRE-1264:


No security manager or any special flags AFAICT. The message is printed because 
of a perm size option that is replaced with metaspace in Java 8. I got rid of 
all output you mentioned but it still hangs: 
https://travis-ci.org/beikov/calcite/builds/278083407
Travis CI kills builds that don't print anything within 10 minutes, but this 
testcase should run through, so it indeed still hangs.
No uses of the mentioned APIs as far as I can see in the code.

I don't understand why this would only be a problem on Linux. Maybe it's a 
concurrency issue?

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173768#comment-16173768
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
Do you use SecurityManager?

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173764#comment-16173764
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
Was Surefire timed out by Travis in the project {{calcite-core}}?
I think the native stream STDOUT was corrupted by printing JVM and GC messages, 
see [1] - how did you print it?.
Please remove all switches like {{-Xlint}}, {{-XX}}, {{-verbose:}}, etc. if you 
are printing some jvm logs by jvm to the console.
Give it a try and let me know then.
Do you use {{FileDescriptor.out}} from Java NIO or some native library printing 
to std/out and std/err, or {{System.exit()}} or {{Runtime.halt()}} or 
ShutdownHook ?

[1]: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option 
MaxPermSize=256m; support was removed in 8.0
Picked up _JAVA_OPTIONS: -Xmx2048m -Xms512m

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172879#comment-16172879
 ] 

Christian Beikov commented on SUREFIRE-1264:


I did, but here it's again.

2.18: https://travis-ci.org/beikov/calcite/builds/277648098
2.18.1: https://travis-ci.org/beikov/calcite/builds/277650952

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172871#comment-16172871
 ] 

Tibor Digana commented on SUREFIRE-1264:


Did you try out Surefire 2.18 and 2.18.1?
What is the result?
Can you try again please?
I am facing a difference in code between 2.18 and 2.19.

On Wed, Sep 20, 2017 at 9:36 AM, Christian Beikov (JIRA) 



> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172855#comment-16172855
 ] 

Christian Beikov commented on SUREFIRE-1264:


Still hanging: https://travis-ci.org/beikov/calcite/builds/277637640

I haven't had the problem on Windows either, just in the Docker container so 
far, but I assume this might be a general problem on linux, though I can't say 
for sure.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172109#comment-16172109
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~cbeikov]
[~christian.beikov]
I do not have this problem on Windows but my tests failed:

{quote}> java.lang.AssertionError: Correlation variable $cor0 should be defined
>   at 
> org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.getCorrelVariableGetter(EnumerableRelImplementor.java:480){quote}

and this is build summary:

{quote}[INFO] Calcite ... SUCCESS [  
6.433 s]
[INFO] Calcite Linq4j  SUCCESS [ 13.526 s]
[INFO] Calcite Core .. FAILURE [01:31 min]
[INFO] Calcite Cassandra . SKIPPED
[INFO] Calcite Druid . SKIPPED
[INFO] Calcite Elasticsearch . SKIPPED
[INFO] Calcite Examples .. SKIPPED
[INFO] Calcite Example CSV ... SKIPPED
[INFO] Calcite Example Function .. SKIPPED
[INFO] Calcite File .. SKIPPED
[INFO] Calcite MongoDB ... SKIPPED
[INFO] Calcite Pig ... SKIPPED
[INFO] Calcite Piglet  SKIPPED
[INFO] Calcite Plus .. SKIPPED
[INFO] Calcite Spark . SKIPPED
[INFO] Calcite Splunk  SKIPPED
[INFO] Calcite Ubenchmark  SKIPPED{quote}

I have a suspicion in commit {{67c06d5876dcabc037de9b5f4cc226beeedcb379}} 
because this touched {{ParallelComputerBuilder}} and you run parallel tests.
Can you please make an experiment?
I think now only you can do it.
Please clone project {{https://github.com/apache/maven-surefire/}} from master 
and revert only changes in class {{ParallelComputerBuilder.java}}. This means 
the line 432 is {{runs.add( runner );}} and you should call it in IF like this:

{code:java}
if ( children != 0 )
{
runs.add( runner );
}
{code}

Save this change and install Surefire project without running tests: {{mvn 
-DskipTests install}}.
Now edit POMs in your project and use another version of Surefire 
{{2.21.0-SNAPSHOT}}.
Run you project with {{no-snapshot-updates}} like this {{mvn -nsu 
-Dsurefire.useFile=false -Dcheckstyle.skip=true test}}.
If the same problem was reproducible before without having Docker, you can 
easily run it on your local machine without Docker. Maybe this would be easier. 
What happens with the test after this?

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Christian Beikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171544#comment-16171544
 ] 

Christian Beikov commented on SUREFIRE-1264:


I am trying to build Apache Calcite on Travis CI with a parallel configuration 
and am consistently running into a hang. I tried various versions of the 
Surefire plugin(1.18, 1.18.1, 1.19.1, 1.20.1) and all seem to have the same 
problem. I did a thread dump after a timeout, but I am not sure what the 
problem is. It seems to be related to this issue. For testing purposes I 
disabled every test but the seemingly problematic parameterized test and in 
fact, it hangs consistently: 
https://travis-ci.org/beikov/calcite/builds/277258873
Without the parallel configuration or when running with a single CPU, the 
problem doesn't seem to happen.

You can run this on a custom VM too. I configured a VirtualBox VM with 2 CPUs 
and 4GB RAM and ran the following commands to execute this locally

{code}
docker run --name travis-debug -dit travisci/ci-garnet:packer-1503972846 
/sbin/init
docker exec -it travis-debug bash -l
su - travis
git clone --branch=travis-test https://github.com/beikov/calcite.git
cd calcite
mvn -Dsurefire.useFile=false -Dcheckstyle.skip=true test
{code}

The exact configuration is

{code}
1
true
both
once
{code}

Here the stack traces..

{code}
"pool-1-thread-2" #13 daemon prio=5 os_prio=0 tid=0x7f0694ff1000 nid=0xc97 
waiting on condition [0x7f067c934000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x80082c78> (a 
java.util.concurrent.Semaphore$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at 
org.apache.maven.surefire.junitcore.pc.ThreadResourcesBalancer.acquirePermit(ThreadResourcesBalancer.java:76)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler.schedule(Scheduler.java:363)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:410)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-1" #12 daemon prio=5 os_prio=0 tid=0x7f0694fef800 nid=0xc96 
waiting on condition [0x7f067ca36000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x80702e60> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"process reaper" #10 daemon prio=10 os_prio=0 tid=0x7f06946c9800 nid=0xc94 
waiting on condition [0x7f067ce6f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x8064cb98> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160555#comment-16160555
 ] 

Hudson commented on SUREFIRE-1264:
--

SUCCESS: Integrated in Jenkins build maven-surefire #1757 (See 
[https://builds.apache.org/job/maven-surefire/1757/])
[SUREFIRE-1264] Some tests can be lost when running in parallel with (tibor17: 
[http://git-wip-us.apache.org/repos/asf/?p=maven-surefire.git=commit=c8724dd2f992625c355e4817a6b7995d3e88fdb4])
* (add) 
surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/jiras/Surefire1264IT.java
* (edit) 
surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/junit4/MockReporter.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreRunListener.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/ATest.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/TestSet.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/ConcurrentRunListener.java
* (add) surefire-integration-tests/src/test/resources/surefire-1264/pom.xml
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/CTest.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/DTest.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/BaseTest.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-1264/src/test/java/com/appnexus/viewability/core/surefireJunitTests/BTest.java
* (edit) 
surefire-providers/surefire-junit47/src/test/java/org/apache/maven/surefire/junitcore/JUnitCoreRunListenerTest.java
* (edit) 
surefire-api/src/main/java/org/apache/maven/surefire/util/internal/TestClassMethodNameUtils.java
* (add) 
surefire-integration-tests/src/test/resources/surefire-621-testCounting-junit3-in-parallel/src/it/java/mho/JUnit4AdapterSuiteTest.java
* (edit) 
surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/jiras/Surefire621TestCountingJunit3InParallelIT.java
* (edit) 
surefire-providers/surefire-junit47/src/test/java/org/apache/maven/surefire/junitcore/TestMethodTest.java
* (edit) 
surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/common/junit4/JUnit4RunListener.java
* (edit) 
surefire-integration-tests/src/test/resources/surefire-621-testCounting-junit3-in-parallel/pom.xml


> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>  Time Spent: 144h
>  Remaining Estimate: 24h
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Valentin Baranov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146830#comment-16146830
 ] 

Valentin Baranov commented on SUREFIRE-1264:


[~tibor17], sorry can't help with this issue right now. There're lots of 
another unexpected activities.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145982#comment-16145982
 ] 

Tibor Digana commented on SUREFIRE-1264:


CI passed all tests successfully:
https://builds.apache.org/job/maven-surefire-pipeline/job/SUREFIRE-1264_2/

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144406#comment-16144406
 ] 

Tibor Digana commented on SUREFIRE-1264:


Flaky tests fail in the build which must be fixed.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144246#comment-16144246
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~Prifiz]
[~valyok]
I have pushed a branch {{SUREFIRE-1264_2}} with a fix.
Can you please build the project on your side and use the version 
{{2.21-SNAPSHOT}} in your project and give me a feedback with your test result.
Please build Surefire project like this:
{{mvn install -DskipTests}}


> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142679#comment-16142679
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~Prifiz]
[~valyok]
I am currently working on {{JUnitCoreRunListener}}, the method 
{{fillTestCountMap}}. It fills out a Map with test classes and TestSet but it 
is not so well written. It should be developed from new beginning and 
{{TestSet#createReportEntry()}} should be changed due to the existence of local 
variable {{isJunit3}} and then I hope this issue will be fixed.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139612#comment-16139612
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~Prifiz]
Definitely the root cause is in {{JUnitCoreRunListener}} . After I removed 
{{isParameterizedRunner( testDesc ) ? testDesc.getClassName() :}} your test 
summary became stable.
There is an issue {{TestClassMethodNameUtils}} with howeve but this is not a 
root cause. We should fix the statement {{return m.matches() ? m.group( 1 ) : 
null}} to {{return m.matches() ? m.group( 1 ) : displayName}}.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Valentin Baranov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137983#comment-16137983
 ] 

Valentin Baranov commented on SUREFIRE-1264:


[~tibor17], unfortunately, I had no time for this during past few days :(

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137273#comment-16137273
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~Prifiz]
[~valyok]
Did you have time to have a look at this issue?
If you remember we spoke about it in the chat of SUREFIRE-1302.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tanguy Le Meur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072573#comment-16072573
 ] 

Tanguy Le Meur commented on SUREFIRE-1264:
--

I confirm that the "bug" with Assert.fail tests has been introduced in 
5593bf560aa8d8ed900fb146176a39ad5b0e42af
Before, the number of test

That being said, before, there was already a bug. So to sum up:
* Before 5593bf560aa8d8ed900fb146176a39ad5b0e42af:
** passing tests lead to omitted tests as described in the current ticket: 
SUREFIRE-1264.
** Assert.fail don't cause this problem
* After 5593bf560aa8d8ed900fb146176a39ad5b0e42af:
** Assert.fail tests lead to omitted tests.
** passing tests work fine and are well counted.
** Assume.assert have the same problem as Assert.fail tests.@Ignore test work 
fine.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tanguy Le Meur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072218#comment-16072218
 ] 

Tanguy Le Meur commented on SUREFIRE-1264:
--

Hi [~tibor17],

I just did a few changes to highlight the issue, I tried to make the issue more 
reproducible.
As I understand the issue lies in the Assert.fail. If a test "fail" then no 
additional test from this class are started (maybe except if they are already 
in queue).
To make it more obvious, I just made all tests to Assert.fail.
I also added a class (DTest) for the same reason. So both threads (out of 2) 
can trigger the issue. (Each thread slot will generally handle 2 classes).
Note also that all is key here to reproduce more easily 
the issue (it also happens with the other configs).
This statements are just guesses, and are not based on actual facts as I was 
not really able to understand the underlying code. This is just based on a ton 
of tries (well of course with some automation :D)

As you suggest, I'll try the different versions to see the output with each 
commit.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071566#comment-16071566
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~tlemeur]
Pls see my previous upload of the ZIP and comment in Jira. Your old test code 
works with the fix 5593bf560aa8d8ed900fb146176a39ad5b0e42af.
It does not work with HEAD of your code. I see you made some changes. Can you 
explain why these changes make random behavior? Locally I used 
{{ConcurrentLinkedQueue}} instead of {{ArrayList}} in your code because of 
multithreading but this did not make stable test result.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071559#comment-16071559
 ] 

Tibor Digana commented on SUREFIRE-1264:


Install Surefire with command {{mvn clean install -DskipTests}}.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071557#comment-16071557
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~tlemeur]
What you changed in the branch? Because now I can reproduce the issue.
Can you checkout the old commit 5593bf560aa8d8ed900fb146176a39ad5b0e42af of 
Surefire and test your branch with that?
If this works then we have to go to HEAD commit by commit and find which one is 
wrong.
We should test Rik Smith's code on every commit of Surefire as well.
Use offline mode {{mvn -o test}} and don't let your local repository to be 
overridden by Maven Central.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tanguy Le Meur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066618#comment-16066618
 ] 

Tanguy Le Meur commented on SUREFIRE-1264:
--

[~tibor17]: that's interesting:

It looks like my branch work fine with surefire 2.20 (public version) but it 
does not with 2.21-SNAPSHOT.
There was a problem before too for sure, but my branch is not showing it.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tanguy Le Meur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066568#comment-16066568
 ] 

Tanguy Le Meur commented on SUREFIRE-1264:
--

[~tibor17]:
Thank you. I updated the git repo accordingly.

1) I added a class and now I think it's even harder to have the branch to not 
show the problem. (99%+ of the time)
2) I'm using the same versions as you. I reproduce on Mac & Linux (I haven't 
tried on Windows).
3) I tried also [~rsmith86] code, and it shown the problem too sometimes 
(roughly 5% of the time).
4) That's not the problem here, but the run order seems not really 
reproducible. But maybe it's only due to the logging order?
As an example
using "alphabetical", I sometimes see CBAD, CABD, CDAB... < I don't know why it 
always start with C
"reversealphabetical": DCBA, CDBA, < It looks to work as I expect.
5) This order thing is not the problem, I reproduce it independentely of the 
order setting. It looked to be dependant only sometimes due to a previous 
version that had different test durations.


> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065477#comment-16065477
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~tlemeur]
I have attached my {{pom.xml}}. See the difference.
Run the command {{mvn -o test}} twice. See {{runOrder}} in my pom.xml. The 
first run does not provide {{power balanced}} order of tests but the second run 
and every next one does. So the first run was (CTest, ATest, BTest) but second 
and the third was (CTest, BTest, ATest). Notice I used Maven 3.5.0 and JDK 8 
u132.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tanguy Le Meur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064858#comment-16064858
 ] 

Tanguy Le Meur commented on SUREFIRE-1264:
--

Hi [~tibor17],

Hmmm, that's surprising that you don't reproduce, whilst it's a bit random, I 
reproduce it more than 75% of the time (so waiting a few execution only show 
the problem rapidly) - all helps a lot here to make it 
happen more frequently.

I fixed the POM in the branch as you suggested. I still reproduce.

Here is a trace of a failing case:
{code}
[INFO] Scanning for projects...
[INFO]
[INFO] 
[INFO] Building main 1.0.0
[INFO] 
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ main ---
[INFO] Deleting /Users/tlemeur/git/surefire-junit-tests/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ main ---
[WARNING] Using platform encoding (US-ASCII actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/Users/tlemeur/git/surefire-junit-tests/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ main ---
[WARNING] File encoding has not been set, using platform encoding US-ASCII, 
i.e. build is platform dependent!
[INFO] Compiling 1 source file to 
/Users/tlemeur/git/surefire-junit-tests/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ 
main ---
[WARNING] Using platform encoding (US-ASCII actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/Users/tlemeur/git/surefire-junit-tests/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ main 
---
[WARNING] File encoding has not been set, using platform encoding US-ASCII, 
i.e. build is platform dependent!
[INFO] Compiling 4 source files to 
/Users/tlemeur/git/surefire-junit-tests/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.21-SNAPSHOT:test (default-test) @ main ---
[WARNING] Missing POM for 
org.apache.maven.surefire:surefire-booter:jar:2.21-SNAPSHOT
[INFO]
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.ATest
Start: ATest.methodA1[p0]
End:   ATest.methodA1[p0]
Start: ATest.methodA1[p1]
End:   ATest.methodA1[p1]
Start: ATest.methodA2[p0]
Start: ATest.methodA2[p1]
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.112 s 
<<< FAILURE! - in com.appnexus.viewability.core.surefireJunitTests.ATest
[ERROR] methodA1[p0](com.appnexus.viewability.core.surefireJunitTests.ATest)  
Time elapsed: 0.112 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p0]
at 
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[ERROR] methodA1[p1](com.appnexus.viewability.core.surefireJunitTests.ATest)  
Time elapsed: 0.112 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p1]
at 
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[INFO] Running com.appnexus.viewability.core.surefireJunitTests.CTest
Start: CTest.methodC1[p0]
End:   CTest.methodC1[p0]
Start: CTest.methodC2[p0]
End:   CTest.methodC2[p0]
Start: CTest.methodC1[p1]
End:   CTest.methodC1[p1]
Start: CTest.methodC2[p1]
End:   CTest.methodC2[p1]
Start: CTest.methodC1[p2]
End:   CTest.methodC1[p2]
Start: CTest.methodC2[p2]
End:   CTest.methodC2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.524 s 
- in com.appnexus.viewability.core.surefireJunitTests.CTest
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.BTest
Start: BTest.methodB2[p0]
End:   BTest.methodB2[p0]
Start: BTest.methodB1[p1]
End:   BTest.methodB1[p1]
Start: BTest.methodB1[p0]
End:   BTest.methodB1[p0]
Start: BTest.methodB2[p1]
End:   BTest.methodB2[p1]
Start: BTest.methodB1[p2]
End:   BTest.methodB1[p2]
Start: BTest.methodB2[p2]
End:   BTest.methodB2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.312 s 
- in com.appnexus.viewability.core.surefireJunitTests.BTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p0]
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p1]
[INFO]
[ERROR] Tests run: 16, Failures: 2, Errors: 0, Skipped: 0
[INFO]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 4.683 s
[INFO] Finished at: 2017-06-27T15:33:49+02:00
[INFO] Final Memory: 17M/169M
[INFO] 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064283#comment-16064283
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~tlemeur]
I changed your POM. I do not see problem. What's wrong?
Do not use plugin in dependencies.
Use {{surefire-junit47}} provider in dependencies embedded within plugin and 
not in project/dependencies; see the documentation [provider for 
plugin|http://maven.apache.org/surefire/maven-surefire-plugin/examples/providers.html].
{code}
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.CTest
Start: CTest.methodC1[p0]
End:   CTest.methodC1[p0]
Start: CTest.methodC2[p0]
End:   CTest.methodC2[p0]
Start: CTest.methodC1[p1]
End:   CTest.methodC1[p1]
Start: CTest.methodC2[p1]
End:   CTest.methodC2[p1]
Start: CTest.methodC1[p2]
End:   CTest.methodC1[p2]
Start: CTest.methodC2[p2]
End:   CTest.methodC2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.301 s 
- in com.appnexus.viewability.core.surefireJunitTests.CTest
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.ATest
Start: ATest.methodA2[p0]
End:   ATest.methodA2[p0]
Start: ATest.methodA1[p0]
End:   ATest.methodA1[p0]
Start: ATest.methodA1[p1]
End:   ATest.methodA1[p1]
Start: ATest.methodA2[p1]
End:   ATest.methodA2[p1]
Start: ATest.methodA2[p2]
End:   ATest.methodA2[p2]
Start: ATest.methodA1[p2]
End:   ATest.methodA1[p2]
[ERROR] Tests run: 6, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.205 s 
<<< FAILURE! - in com.appnexus.viewability.core.surefireJunitTests.ATest
[ERROR] methodA1[p0](com.appnexus.viewability.core.surefireJunitTests.ATest)  
Time elapsed: 0.105 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p0]
at 
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[ERROR] methodA1[p1](com.appnexus.viewability.core.surefireJunitTests.ATest)  
Time elapsed: 0.1 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p1]
at 
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[ERROR] methodA1[p2](com.appnexus.viewability.core.surefireJunitTests.ATest)  
Time elapsed: 0.1 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p2]
at 
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[INFO] Running com.appnexus.viewability.core.surefireJunitTests.BTest
Start: BTest.methodB1[p0]
End:   BTest.methodB1[p0]
Start: BTest.methodB2[p0]
End:   BTest.methodB2[p0]
Start: BTest.methodB1[p1]
End:   BTest.methodB1[p1]
Start: BTest.methodB2[p1]
End:   BTest.methodB2[p1]
Start: BTest.methodB1[p2]
End:   BTest.methodB1[p2]
Start: BTest.methodB2[p2]
End:   BTest.methodB2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.206 s 
- in com.appnexus.viewability.core.surefireJunitTests.BTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p0]
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p1]
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p2]
[INFO]
[ERROR] Tests run: 18, Failures: 3, Errors: 0, Skipped: 0
[INFO]
{code}

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061860#comment-16061860
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~tlemeur]
First I have to finish another pending issue and then I will dig into this. I 
will inform you.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tanguy Le Meur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061108#comment-16061108
 ] 

Tanguy Le Meur commented on SUREFIRE-1264:
--

Hi [~tibor17],

Thank a lot for your fix. I did some extensive testing on the 2.21-SNAPSHOT 
version, and this indeed improves a lot. For example, I was not able to 
reproduce the error using [~jderrien] branch.

However if a test fails (Assert.fail), later tests can still be lost.

I made a branch here to show the problem:
https://github.com/tanguylemeur/surefire-junit-tests/tree/failure-2.21-snapshot-SUREFIRE-1264

If it would be ok, then this command should never end:
{code}time while true; do mvn clean  test > last.log ; tail -25 last.log ; if [ 
"$(grep -c 'Tests run: 18' last.log)" == "0" ]; then break; fi ; done{code}

Actually if you look at the trace, it seems that the test count varies 100% of 
the time if it starts with the class "ATest". As far I understand, after a test 
fails, all tests from this class are not executed anymore.

Let me know if you need more information, or if you want me to open a separate 
bug?

Thank you!

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Rik Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004097#comment-16004097
 ] 

Rik Smith commented on SUREFIRE-1264:
-

[~tibor17]

With our test suite i find it difficult to tell whether it is now completely 
accurate, but the results are looking very good. So far I have not found an 
unexpected difference in the test runs from  the few last days.

Thanks a lot for your effort!

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997565#comment-15997565
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~rsmith86]
Try to use Surefire plugin version {{2.21-SNAPSHOT}} and plugin management URL:

http://repository.apache.org/snapshots/

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997556#comment-15997556
 ] 

Hudson commented on SUREFIRE-1264:
--

SUCCESS: Integrated in Jenkins build maven-surefire #1701 (See 
[https://builds.apache.org/job/maven-surefire/1701/])
[SUREFIRE-1264] Some tests can be lost when running in parallel with (tibor17: 
[http://git-wip-us.apache.org/repos/asf/?p=maven-surefire.git=commit=5593bf560aa8d8ed900fb146176a39ad5b0e42af])
* (add) 
surefire-integration-tests/src/test/resources/surefire-1082-parallel-junit-parameterized/src/test/java/jiras/surefire1082/Jira1264Test.java
* (edit) 
surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/jiras/Surefire1082ParallelJUnitParameterizedIT.java
* (edit) 
surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreRunListener.java
* (edit) 
surefire-api/src/main/java/org/apache/maven/surefire/util/internal/TestClassMethodNameUtils.java
* (edit) 
surefire-integration-tests/src/test/resources/surefire-1082-parallel-junit-parameterized/src/test/java/jiras/surefire1082/Jira1082Test.java
* (edit) 
surefire-integration-tests/src/test/java/org/apache/maven/surefire/its/fixture/TestFile.java


> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Rik Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15640150#comment-15640150
 ] 

Rik Smith commented on SUREFIRE-1264:
-

[~tibor17] I am facing the same issue in a project. We also use Parameterized 
tests and run them in parallel (suitesAndClasses) with a fork. I also created a 
simple project to replicate this issue:
https://github.com/riksmith/SUREFIRE-1264-demo

The actual project has a much more complex setup (including a custom junit 
runner code), but I hope a fix for this sample project will also fix our actual 
project.

Perhaps good to know:
If you run the sample project with parallel: classes the issue does not occur. 
If you run the sample project with forkCount=0 the issue does not occur.

I think it is some kind of threading issue. I tried looking into the code but I 
find it quite difficult to understand and I can't get it to debug properly.

It is quite an important issue for us because this means that builds are 
sometimes succeeding while actually there are tests failing!

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Jean-Luc Derrien (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523278#comment-15523278
 ] 

Jean-Luc Derrien commented on SUREFIRE-1264:


[~tibor17], sorry I won't be able to provide a patch. I don't understand the 
underlying code and don't have the required skills to fix the issue.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15520650#comment-15520650
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~jderrien]
Please try to help us to fix it and open a pull request in GitHub with your 
patch.
Thx.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Jean-Luc Derrien (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422967#comment-15422967
 ] 

Jean-Luc Derrien commented on SUREFIRE-1264:


[~tibor17], I've modified my branch 
[simple-1|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] 
accordingly.

I still reproduce the problem with {{ConcurrentLinkedQueue}}. It seems to be a 
reporting error since all the 12 files are created even when the output reports 
11 tests:

{noformat}
---
 T E S T S
---
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
sleeptime = 1 => start
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
[p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
[p2]

Results :

Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 7.315 s
[INFO] Finished at: 2016-08-16T18:06:26+02:00
[INFO] Final Memory: 16M/168M
[INFO] 
{noformat}

{noformat}
$ ls target/*.log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p2].log
{noformat}


Note: I also reproduce the problem on Linux with different version of Java and 
Maven:
{noformat}
aiantis@s30:~/tmp/surefire-junit-tests$ mvn --version
Warning: JAVA_HOME environment variable is not set.
Apache Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200)
Java version: 1.7.0_95
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux" version: "3.14.32--std-ipv6-64" arch: "amd64" Family: 
"unix"
{noformat}

In this case, the output is slightly different (10 tests only, and the output 
shows the {{=> stop}} logs correctly when the problem occurs, which is not the 
case with my other setup):
{noformat}
---
 T E S T S
---
Running [p2]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
[p2]
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
sleeptime = 2 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
sleeptime = 5 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
sleeptime = 1 => stop
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.002 sec - in 
[p2]

Results :

Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD SUCCESSFUL
[INFO] 
[INFO] Total time: 7 seconds
[INFO] Finished at: Tue Aug 16 17:32:18 CEST 2016
[INFO] Final Memory: 30M/478M
[INFO] 
{noformat}

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ 

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420277#comment-15420277
 ] 

Tibor Digana commented on SUREFIRE-1264:


It might be an issue that {{parameterList()}} uses {{ArrayList}} which is not 
threadsafe. Try to use {{ConcurrentLinkedQueue}} instead.

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests

[Tibor Digana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420276#comment-15420276
 ] 

Tibor Digana commented on SUREFIRE-1264:


[~jderrien]
I don't use MAC so it is quite difficult to reproduce it for me. Can you try to 
create individual files for every cls-method-parameter on the file system to 
see if it is real or just displaying issue?

> Some tests can be lost when running in parallel with parameterized tests
> 
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
>  Issue Type: Bug
>  Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>
> Hello,
> It appears some tests can be lost when using the parallel mode with 
> parameterized tests. Here is a [small 
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've 
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on 
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ 
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - 
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
>  T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - 
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - 
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - 
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - 
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - 
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master 
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's 
> even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)