[
https://issues.apache.org/jira/browse/HADOOP-19978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111772#comment-18111772
]
ASF GitHub Bot commented on HADOOP-19978:
-----------------------------------------
joseluisll opened a new pull request, #8720:
URL: https://github.com/apache/hadoop/pull/8720
### Description of PR
Upgrades `maven-surefire-plugin` from 3.5.3 to 3.6.0, plus the two changes
the new version needs.
3.6.0 is the first release that registers surefire's `listener` property as
a JUnit Platform `TestExecutionListener` (SUREFIRE-1639). Eight poms already
carry that property for `TimedOutTestsListener`, and on 3.5.3 it does nothing
at all. This bump is what lets HADOOP-19964's listener bind, so a timed-out
test prints a thread dump again. Until 19964 lands the listener stays dormant —
this change is what unblocks it.
Two changes come with the bump:
- **`java.net.preferIPv4Stack` moves into `maven-surefire-plugin.argLine`.**
As a `systemPropertyVariable` it is applied only after the forked JVM has
started. 3.6.0 initialises the fork's networking before applying those
properties, so the flag arrives too late and the fork falls back to IPv6,
turning 5 of 7 CI test jobs red; 3.5.3 applies them before any networking class
loads, which is why the same configuration works there. As an argLine entry the
flag is a real JVM startup argument and cannot be late. All the `<argLine>`
overrides in the tree interpolate the property and maven-failsafe-plugin shares
it, so it propagates everywhere. `hadoop-registry` carried a second copy of the
same ineffective setting; removed.
- **`hadoop-common`'s test-jar added to `hadoop-yarn-services-core` and
`hadoop-yarn-applications-catalog-webapp`.** Both inherit the listener property
from `hadoop-yarn` but resolve only `hadoop-common`'s main jar, so the listener
class cannot be loaded there and surefire silently drops it.
### Known impact of 3.6.0, needs a decision
3.6.0 also includes SUREFIRE-823, which **decouples `-DskipTests` from
maven-failsafe-plugin**. On 3.5.3 `-DskipTests` skips both surefire and
failsafe; on 3.6.0 it skips surefire only, and failsafe runs. `-DskipITs` still
skips integration tests.
For most of the tree this is invisible — `hadoop-aws`, `hadoop-azure`,
`hadoop-gcp` and `hadoop-bos` gate their ITs on `skipITs` via the presence of
`auth-keys.xml`, so they stay skipped.
`hadoop-client-modules/hadoop-client-integration-tests` does not, and its
integration tests now run under `mvn install -DskipTests`, the command in
BUILDING.txt.
This PR does not address that yet. Options are to bind failsafe's skip to
`skipTests` in `hadoop-project`, to update BUILDING.txt and CI to pass
`-DskipITs`, or to handle it in a follow-up JIRA. Happy to take direction from
reviewers.
### How was this patch tested?
WSL2 Ubuntu 24.04, OpenJDK 17, Maven 3.9.16. Full reactor build clean.
Full suites on the modules this touches, all green: `hadoop-registry` (164
tests), `hadoop-yarn-services-core` (145),
`hadoop-yarn-applications-catalog-webapp` (5).
`hadoop-common` full suite run against unpatched trunk at the same commit as
a control: every failure reproduces on the baseline, and the two that did not
pass on rerun. No new failures.
The tests that fail when 3.6.0 is bumped without the argLine fix all pass:
`TestDNS`, `TestNameNodeRpcServer`, `TestBlockRecovery2`, `TestWebHdfsTokens`,
`TestHSAdminServer`.
One reporting change to be aware of: 3.6.0 counts assumption-skipped tests
that 3.5.3 reported as zero, which raises `hadoop-common`'s skip count by 40
(two OpenSSL codec classes).
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.
> Upgrade maven-surefire-plugin to 3.6
> ------------------------------------
>
> Key: HADOOP-19978
> URL: https://issues.apache.org/jira/browse/HADOOP-19978
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
> Reporter: Jose Luis López
> Priority: Major
>
> h3. Why
> HADOOP-19964 reimplements TimedOutTestsListener as a JUnit Platform
> TestExecutionListener, so a timed-out test prints a thread dump again. The
> {{listener}} provider property that eight poms already carry binds to it only
> once Surefire accepts platform listeners there — SUREFIRE-1639, shipped in
> maven-surefire-plugin 3.6.0. Until Hadoop moves to that version the listener
> is dormant.
> h3. What blocks the upgrade
> Bumping {{maven-surefire-plugin.version}} from 3.5.3 to 3.6.0 turns 5 of 7 CI
> test jobs red: common, mr, hdfs-rbf, hdfs-other, hdfs-slow. Controlled against
> another branch built the same morning on the same runners with 3.5.3, which
> was 7/7 green. Every failure is one fault — the forked JVM preferring IPv6:
> {code:java}
> TestNameNodeRpcServer.testNamenodeRpcBindAny expected: <0.0.0.0> but was:
> <0:0:0:0:0:0:0:0>
> TestHSAdminServer not a valid host:port authority: 0:0:0:0:0:0:0:0:33745
> TestBlockRecovery2 not a valid host:port authority: 0:0:0:0:0:0:0:0:39677
> TestDNS ArrayIndexOutOfBounds Index 3 out of bounds for length 1
> TestWebHdfsTokens Connect localhost:44787: Connection refused
> {code}
> hadoop-project.xml sets {{java.net.preferIPv4Stack=true}} in surefire's
> {{{}systemPropertyVariables{}}}. Those are applied by the forked booter, not
> as
> {{-D}} at JVM startup — in 3.5.3 and 3.6.0 alike. Class-load traces
> ({{{}-Xlog:class+load{}}}) show SystemPropertyManager running at index 699
> under
> 3.5.3 and 736 under 3.6.0; in that gap 3.6.0 loads {{sun.nio.ch.IOUtil}} and
> {{{}sun.nio.ch.SocketChannelImpl{}}}, where 3.5.3 loads no networking classes
> at
> all. 3.6.0 initialises the fork's networking before the property lands, so the
> flag is read too late.
> Reported upstream as apache/maven-surefire#3456.
> h3. Changes
> * Bump {{maven-surefire-plugin.version}} to 3.6.0.
> * Move {{-Djava.net.preferIPv4Stack=true}} into
> {{maven-surefire-plugin.argLine}} so it is a real JVM startup argument. All
> 22 {{<argLine>}} overrides in the tree already interpolate that property, and
> maven-failsafe-plugin shares it, so it propagates everywhere. No
> property-based fix can work: Surefire initialises networking before applying
> any of them.
> * Add hadoop-common's test-jar in test scope to hadoop-yarn-services-core and
> hadoop-yarn-applications-catalog-webapp. Both inherit the {{listener}}
> property from hadoop-yarn but depend only on the main jar, so with the
> listener live Surefire cannot resolve the class and the fork dies before any
> test runs. Verified: without the dependency the module reports "Tests run: 0"
> and the fork terminates; with it, tests pass.
> h3. Notes
> Only {{java.net.preferIPv4Stack}} is affected. Checking the other
> systemPropertyVariables against the same criterion — is anything initialised
> before the properties are applied? — {{{}java.security.egd{}}},
> {{{}java.security.krb5.conf{}}}, {{{}hadoop.log.dir{}}},
> {{{}hadoop.tmp.dir{}}},
> {{test.build.*}} and {{require.test.libhadoop}} are all read later and are
> unaffected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
