[
https://issues.apache.org/jira/browse/SUREFIRE-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645390#comment-17645390
]
Elliotte Rusty Harold commented on SUREFIRE-2087:
-------------------------------------------------
OK, maybe I see it. The issue is specifically with parameterized tests. Note
that "Instead of failing TestA, it's marked as flaky giving an overall pass for
the test suite." is still WAI. That is, Test A should be marked as a pass, not
failed.
> rerunFailingTestsCount incorrectly marks failed parametrized test as pass
> -------------------------------------------------------------------------
>
> Key: SUREFIRE-2087
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2087
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M4
> Reporter: Ramanan Rajendran
> Priority: Critical
>
> rerunFailingTestsCount is not working as expected with junit5
> `@ParameterizedTest`
> If 1st iteration of the test passes and the 2nd one fails, it still marks the
> test as flaky passed.
> Eg: In the following run, iteration 1 with input A passed but iteration 2
> with input B failed. Iteration 2 was rerun 2 times and failed in all the
> attempts. Instead of failing TestA, it's marked as flaky giving an overall
> pass for the test suite. I believe it's using the Run 1 PASS as a gate to
> mark the whole test as a pass.
>
> [INFO] Run 1: PASS
> [ERROR] Run 2: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 3: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 4: TestA.iteration2:101 1 expectation failed.
> <400> doesn't match actual status code <401>.
>
> Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
