Thanks Westom for bringing this up. It's good to have everyone's attention.
As mentioned in the thread that Weston refers to [1], a few projects are leaping Java 17 (Spark 4), and the rest of them are looking at each other to move first :) This was also discussed back in February on the Iceberg Devlist (both have Parquet and Arrow as a dependency) [2]. I also started a related thread on the Parquet devlist in February [3]. Kind regards, Fokko [1] https://lists.apache.org/thread/65vqpmrrtpshxo53572zcv91j1lb2y8g [2] https://lists.apache.org/thread/ntrk2thvsg9tdccwd4flsdz9gg743368 [3] https://lists.apache.org/thread/kttwbl5l7opz6nwb5bck2gghc2y3td0o Op vr 31 mei 2024 om 18:38 schreef Steve Loughran <[email protected]>: > 1. Hadoop doesn't use arrow. > 2. The Hadoop team would love to drop java 8 and in the last release > said "will happen soon" > 3. all the client stuff is happy java17+, it's just some of the server > side stuff which is a bit of pain point. > 4. Hadoop 3.2.x is needed to move beyond java8. So if you commit to > saying "java11+" now is the time to drop that hadoop 2 profile and build > against the most recent hadoop 3.2 build. > 5. going for 3.3.0+ would be even better, given that it is the oldest > maintained branch (CVEs, bug fixes) > > > > On Mon, 27 May 2024 at 09:17, Gábor Szádovszky <[email protected]> wrote: > > > Thanks a lot Weston for bringing this up. > > > > Last time we discussed a potential java upgrade, Hadoop was the one not > > allowing us to do so. Hadoop is still on java 8. > > If we want to keep Arrow on the latest version, we will need to upgrade > to > > java 11. In this case we won't be able to support Hadoop with our later > > releases. > > Another option is to keep our source at java 8 level and provide > artifacts > > with java 8 and java 11 builds as well. Or at least build parquet-arrow > > with the latest Arrow version on java 11. Do we know if Hadoop uses Arrow > > and/or parquet-arrow? > > > > What do the community think about this? > > > > Cheers, > > Gabor > > > > Weston Pace <[email protected]> ezt írta (időpont: 2024. máj. 24., > P, > > 17:51): > > > > > Arrow is moving towards dropping support for Java 8 in the next release > > > (details and justification can be found in [1]). The parquet-java > > > repository currently uses the latest version of Arrow (which makes me > > think > > > parquet-java would upgrade) and has CI tests that run on Java 8 (which > > > makes me think parquet-java still supports Java 8). > > > > > > It appears the recommendation, should you wish to continue supporting > > Java > > > 8, is to use Arrow version 16 on releases that need Java 8 support. > > > > > > Please direct any feedback or concerns to [1]. > > > > > > [1] https://lists.apache.org/thread/65vqpmrrtpshxo53572zcv91j1lb2y8g > > > > > >
