gnodet commented on PR #3352: URL: https://github.com/apache/maven-surefire/pull/3352#issuecomment-5077361944
## CI Analysis — Maven 4 IT Failures All **Maven 3** cells pass ✅ across all OS/JDK combinations. The **Maven 4** cells show ~56 failing ITs out of 751 (~92.5% pass rate). Analysis of the failures: ### Exit Code 2 Failures (20 tests) **Affected:** `ForkCountIT` (8), `ForkCountMultiModuleIT` (4), `ForkCountTestNGIT` (8) **Root cause:** A [known bug in Maven 4.0.0-rc-5's `bin/mvn` script](https://github.com/apache/maven/commit/e6a9c4216b). The rc-5 script uses `eval exec "$cmd"` which interprets `${...}` patterns as shell variables. These tests pass `-DtestProperty=testValue_${surefire.threadNumber}_${surefire.forkNumber}` as a CLI argument. On `dash` (Ubuntu's `/bin/sh`), dots in variable names trigger `Bad substitution` → exit code 2. **Status:** Already fixed in Maven's main branch ([e6a9c4216b](https://github.com/apache/maven/commit/e6a9c4216b), 2026-05-19) and 4.0.x branch ([8f7bdee4c4](https://github.com/apache/maven/commit/8f7bdee4c4), 2026-05-20), but not yet released. **Not fixable in surefire** — the bug is in Maven's launcher script. ### Other Maven 4 Failures (~36 tests) Genuine Maven 4 behavioral differences: - **FailFastJUnit4IT** (7): `Reports directory is missing` — Maven 4 build completes but surefire-reports directory not generated - **RunOrderIT / RunOrderParallelForksIT** (14): Test execution order assertions fail — Maven 4 may handle run ordering differently - **Various Surefire*IT** (~15): Assertion failures on test counts, missing log text, classloader differences These are separate Maven 4 compatibility issues that need follow-up investigation. ### Summary | Category | Tests | Fixable Here? | |----------|-------|---------------| | Maven 3 (all cells) | ✅ All pass | — | | Maven 4 exit code 2 | 20 failures | ❌ Maven script bug (fixed in unreleased main) | | Maven 4 behavioral | ~36 failures | ❌ Need separate follow-up PRs | | Maven 4 passing | ~695 passing | ✅ | The `isMaven4()` detection + forked mode workaround resolved the original `NoSuchMethodException` from `maven-verifier 2.0.0-M1`'s incompatible embedded mode. The remaining failures are inherent Maven 4 compatibility gaps that can be tracked separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
