I would like to share the current progress on Hadoop JDK17. 1. Junit5 Upgrade
We have completed the migration of unit tests to Junit5 across the project and removed Junit4 dependencies wherever possible. In addition, we configured an Import Rule to ensure that Junit4 references are not reintroduced. However, some Junit4 dependencies still remain because certain third-party plugins continue to rely on Junit4 components ( see YARN-11872: TestAppCatalogSolrClient has a Junit4 transitive dependency <https://issues.apache.org/jira/browse/YARN-11872>). We will need to wait until these plugins are upgraded before fully removing Junit4. 2. Unit Test Issues Several failing unit tests remain, mainly in the YARN module, including RM crash tests and other errors. There are also issues reported with the Jesery2 Client (see mailing list thread <https://lists.apache.org/thread/y70j1wqj00rhon4hxjkws6k55514sbfo>). I have set up a complete local environment to run the full test suite and will continue to follow up on these issues with high priority. 3. JDIFF Issues and Progress JDIFF is a long-standing plugin that has not been updated since JDK8. Thanks to the efforts of Hualong Zhang, JDIFF can now be successfully compiled under JDK17. He also updated the custom annotations in hadoop-annotations, enabling jdiff to generate difference files under JDK17. I assisted in reviewing part of the code changes and consider them reasonable. Hualong is currently on leave and is expected to submit these changes to the community next week. Additionally, we identified missing files in some versions: hadoop-3.4.0 is missing Apache_Hadoop_Common_3.4.0.xml; hadoop-3.4.1 and hadoop-3.4.2 did not upload jdiff comparison files to the trunk branch. 4. JDK17 Validation Pipeline With the assistance of Gautham, we have configured a Jenkins JDK17 validation pipeline. I will run a full validation on the trunk branch as soon as possible, and if no issues are found, we plan to officially switch the trunk branch build to JDK17 within the next 1–2 weeks. Best Regards, Shilun Fan On Fri, Jul 11, 2025 at 3:16 PM Ayush Saxena <[email protected]> wrote: > Hi Shilun, > > Thanks for initiating this discussion and for all the efforts you're > putting into the 3.5.0 release — much appreciated! > > > Ongoing Support for the Hadoop 3.4 Branch > > It’s not just 3.4 — we currently have active release lines for 2.10, > 3.2, and 3.3 as well. The officially designated EOL release lines are > listed here [1]. Personally, I think it’s time we consider marking 3.2 > and 3.3 as EOL. However, I would be cautious about doing the same for > 3.4 immediately after the 3.5.0 release. > > 3.5.0 brings substantial changes, including the JDK migration, which > might make immediate adoption difficult for some users. I suggest we > keep 3.4.x as the sole active JDK 8-compatible line for a while > longer. We could maintain it with selective cherry-picks for critical > issues and only roll a release if there’s significant demand. We can > always revisit this decision a few months after 3.5.0 is out. > > While many projects have moved to newer JDK versions, a few downstream > consumers still rely on JDK 8 — that’s something we should factor in. > > > Preparation for Hadoop 3.5 Release > > Great to see progress on this front. My responses inline: > > > Should we set up a new CI pipeline for JDK 17 on the trunk branch? > > Yes, since we're bumping the compilation version to 17, our regular CI > should move entirely to JDK 17. > > > Should we also add support for JDK 21 in CI? > > I don’t think running full test suites on both JDK 17 and 21 is > practical. I propose we follow a similar approach as we did for JDK > 11: > > As of the current code compilation targets JDK 17 only, the --release > 17 flag is already hardcoded [2], so the generated bytecode is always > JDK 17, regardless of whether you build with JDK 21, So we can > configure pre-commit jobs via Yetus ensure multi-JDK compilation > doesn’t break and a separate daily CI job can verify runtime > compatibility on JDK 21 running tests. > > > If JDK 21 support is added, can we remove the existing JDK 11 pipeline? > Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline. > JDK 11 can be dropped. And moreover if you bump the version at [3] to > 17, JDK-11 won't compile only. > > > Operating System Support Strategy > > +1 to all three of your proposals. Maintaining outdated OS images is > tedious and brings little value. It’s better to validate against > newer, actively supported distros and ensure a smooth experience > there. > > > Release Strategy Under Multi-JDK Builds > > I don't recall any active discussions around multi-JDK release > variants. As mentioned earlier, we would be compiling with JDK 17 as > the version at [3] would change to 17, and that should be the only > binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17. > We can document how users can build from source with higher JDKs if > needed. > > In essence, the official release is the source. Binaries are > convenience artifacts — and should reflect our declared compilation > baseline. > > The other sub-questions under this section become moot in light of the > above, so skipping them for brevity. > > > Just my thoughts — I’m open to adjusting the plan based on what the > broader community agrees upon. > > Best, > Ayush > > [1] > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > [2] > https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769 > [3] > https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205 > > > > On Fri, 11 Jul 2025 at 11:14, slfan1989 <[email protected]> wrote: > > > > Dear community, > > > > I'd like to initiate a discussion on the future release and support > strategy for Hadoop. This includes a few important topics that I believe we > should align on: > > > > 1. Ongoing Support for the Hadoop 3.4 Branch > > > > As it stands, Hadoop 3.4 is the last release line that supports Java 8. > It is expected that Java 8 compilation support will be officially removed > from the trunk branch after a community vote. > > At the same time, the trunk branch is increasingly diverging from the > 3.4 line due to the adoption of JUnit 5 and new features based on JDK17 > syntax. As a result, future bug fixes and backports will become > significantly more difficult. > > Given this, I’d like to ask: after the release of hadoop-3.4.2, does the > community plan to continue maintaining and releasing hadoop-3.4.3 or other > 3.4-series versions? > > > > 2. Preparation for Hadoop 3.5 Release > > > > With the JDK17 support work nearing completion, it’s time to start > planning for the Hadoop 3.5 release. Specific questions for discussion > include: > > > > Should we set up a new CI pipeline for JDK17 on the trunk branch? > > > > Should we also add support for JDK21 in CI? > > > > If JDK21 support is added, can we remove the existing JDK11 pipeline? > > > > 3. Operating System Support Strategy > > > > Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have > already reached or are approaching end-of-life (EOL). > > Should we adjust our support policy for these systems? For example: > > > > Officially deprecate EOL systems; > > > > Provide documentation-only support for compilation; > > > > Remove outdated base images from Dockerfiles. > > > > 4. Release Strategy Under Multi-JDK Builds > > > > As we prepare to support JDK11, JDK17, and JDK21, we also need a clear > strategy for future releases. Key questions include: > > > > Should we publish multiple build variants like: hadoop-3.5.0-jdk11, > hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21? > > > > What naming convention should we adopt for these variants? > > > > For Maven Central artifacts, should we distinguish by JDK version or > consider a unified approach? > > > > Additional Context and Suggestions from My Side: > > > > The JDK17 compatibility work is being tracked under HADOOP-17177. I am > in the process of reviewing and linking all relevant JIRAs—both completed > and pending—to this umbrella JIRA, and will share an overview in the coming > week. > > > > Regarding the JUnit 5 migration: > > Based on our experience, JUnit 5 introduces significant differences > compared to JUnit 4. It fundamentally changes how tests are written and > run. My proposed upgrade path is as follows: > > > > Complete the migration to JUnit 5.8.2 (we are still working on HDFS and > Hadoop-Azure and aim to finish in the next two weeks); > > > > Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS, > Common, etc.); > > > > Upgrade to JUnit 5.13.2 (as Steve suggested); > > > > Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed > use of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is > fully adopted; > > > > Upgrade the Maven version to 3.9.0 for better compatibility and plugin > support. > > > > Once the JUnit 5 migration is complete, we will conduct a thorough test > audit to ensure everything runs correctly and mark unstable tests for > future attention. > > > > Regarding legacy OS support, I recommend a "lightweight" approach: > > > > Remove unsupported base images from our Dockerfiles or upgrade them to > newer versions; > > > > Provide documentation to help users build on older systems; > > > > But formally discontinue full support for EOL systems. > > > > I’d appreciate hearing your thoughts on these topics. Looking forward to > a fruitful discussion! > > > > Best regards, > > Shilun Fan. > > > > > > On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <[email protected]> wrote: > >> > >> Hi, Chris > >> > >> Thank you very much for your efforts in driving the JDK17 upgrade. I > will continue to follow up on the JUnit5 migration, and I hope we can > successfully complete this upgrade together. > >> > >> Best Regards, > >> Shilun Fan. > >> > >> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <[email protected]> wrote: > >> > >> > Hello everyone, > >> > >> > I've mentioned in other forums that I'd volunteer as release manager > for > >> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd > like to > >> > get more specific about release planning and a potential target date. > >> > >> > It can be easy to get lost in the nested historical JIRAs for Java 17 > >> > support [1]. I'd like to start by summarizing my understanding of > work done > >> > and work remaining for Java 17 as well as other significant features > the > >> > community wants to ship in 3.5.0: > >> > >> > - We've completed the Jersey upgrade [2]. > >> > - We've made significant progress on the JUnit 5 upgrade [3], and we > >> > have a list of several PRs remaining to be reviewed and committed > [4]. > >> > - There is some sun.misc usage that needs alternate implementations > [5] > >> > [6]. There are PRs, but they haven't had any recent activity. > >> > - Once these issues are settled, we need to (re-)introduce a Java 17 > >> > build profile [7]. > >> > - I and a few others are also interested in trying to target the GCS > >> > FileSystem implementation for 3.5.0 [8]. > >> > >> > Does this look accurate? Did I miss anything important for 3.5.0, > either > >> > for Java 17 or other features? > >> > >> > Chris Nauroth > >> > >> > [1] https://issues.apache.org/jira/browse/HADOOP-17177 > >> > [2] https://issues.apache.org/jira/browse/HADOOP-15984 > >> > [3] https://issues.apache.org/jira/browse/HADOOP-14693 > >> > [4] > https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756 > >> > [5] https://issues.apache.org/jira/browse/HADOOP-19329 > >> > [6] https://issues.apache.org/jira/browse/HADOOP-19334 > >> > [7] https://issues.apache.org/jira/browse/HADOOP-19298 > >> > [8] https://issues.apache.org/jira/browse/HADOOP-19343 >
