1. Hadoop doesn't use arrow. 2. The Hadoop team would love to drop java 8 and in the last release said "will happen soon" 3. all the client stuff is happy java17+, it's just some of the server side stuff which is a bit of pain point. 4. Hadoop 3.2.x is needed to move beyond java8. So if you commit to saying "java11+" now is the time to drop that hadoop 2 profile and build against the most recent hadoop 3.2 build. 5. going for 3.3.0+ would be even better, given that it is the oldest maintained branch (CVEs, bug fixes)
On Mon, 27 May 2024 at 09:17, Gábor Szádovszky <[email protected]> wrote: > Thanks a lot Weston for bringing this up. > > Last time we discussed a potential java upgrade, Hadoop was the one not > allowing us to do so. Hadoop is still on java 8. > If we want to keep Arrow on the latest version, we will need to upgrade to > java 11. In this case we won't be able to support Hadoop with our later > releases. > Another option is to keep our source at java 8 level and provide artifacts > with java 8 and java 11 builds as well. Or at least build parquet-arrow > with the latest Arrow version on java 11. Do we know if Hadoop uses Arrow > and/or parquet-arrow? > > What do the community think about this? > > Cheers, > Gabor > > Weston Pace <[email protected]> ezt írta (időpont: 2024. máj. 24., P, > 17:51): > > > Arrow is moving towards dropping support for Java 8 in the next release > > (details and justification can be found in [1]). The parquet-java > > repository currently uses the latest version of Arrow (which makes me > think > > parquet-java would upgrade) and has CI tests that run on Java 8 (which > > makes me think parquet-java still supports Java 8). > > > > It appears the recommendation, should you wish to continue supporting > Java > > 8, is to use Arrow version 16 on releases that need Java 8 support. > > > > Please direct any feedback or concerns to [1]. > > > > [1] https://lists.apache.org/thread/65vqpmrrtpshxo53572zcv91j1lb2y8g > > >
