I wanted to share some status updates: I do have a list of changes to build Arrow with a recent version of Java using the `--release` plugin option of javac. The full change is available at: https://github.com/laurentgo/arrow/pull/3
Here are roughly how the changes are divided: * Change access to MemoryUtil.UNSAFE to private, and replace direct invocation of sun.misc.Unsafe methods with MethodHandle instances. The reason for this change is that sun.misc.Unsafe class is not available when using `--release` option (see https://bugs.openjdk.org/browse/JDK-8206937) * Update some maven plugin dependencies to latest versions like maven-compiler-plugin * Add a new profile to allow for cross-version testing. The profile is enabled by using `-Darrow.test.jdk-version=<jdk version to use>` where version can be 1.8, 11, 17, ... Next steps: integrate the change with Arrow CI/CD environment (which is why the change was not posted as a pull request against apache/arrow repository yet). The main issue is that linux builds do not happen directly within the GH runner (like macos or windows build) but inside a container. Unfortunately there's only one JDK version available within the container, so my next step would need to find a way to extend the image or create a new one. If people have other options in mind, I'm also all ear. Laurent On Wed, Apr 10, 2024 at 9:07 AM Laurent Goujon <laur...@dremio.com> wrote: > I can give it a try for sure > > On Fri, Apr 5, 2024 at 10:26 AM Dane Pitkin <d...@voltrondata.com.invalid> > wrote: > >> I think we can revisit the discussion soon for dropping Java 8 altogether, >> since Spark will release 4.0 in ~June supporting Java 17+ at runtime. >> >> I'm curious how big of an effort it would be to get your proposal >> implemented. Would you be willing to draft a PR so we can see what types >> of >> changes are necessary? >> >> On Wed, Apr 3, 2024 at 8:05 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >> > Yes, correct for language features. My point was more that we can >> > decide on a major Arrow version upgrading the target language version. >> > That's what I meant by "consensus". >> > >> > Regards >> > JB >> > >> > On Tue, Apr 2, 2024 at 5:55 PM Laurent Goujon >> > <laur...@dremio.com.invalid> wrote: >> > > >> > > At code level we need to separate language features from library >> > features? >> > > It should be possible to leverage memory API for example through >> > reflection >> > > and/or multi-release jar files, but record is a language feature and >> it >> > > would not possible to use it without targeting java 17 at the source >> > level. >> > > >> > > Laurent >> > > >> > > On Tue, Apr 2, 2024 at 1:40 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> > wrote: >> > > >> > > > Hi Laurent >> > > > >> > > > It makes sense to me. I started this "move" (on the plugin side of >> the >> > > > thing) as part of the reproducible build effort. >> > > > >> > > > At code level, I think it would be great to leverage some features >> > > > from Java 17+ (I'm thinking about record, memory API, etc). >> > > > I would be more than happy to help on this as soon as we have a >> > consensus. >> > > > >> > > > Thanks, >> > > > Regards >> > > > JB >> > > > >> > > > On Mon, Apr 1, 2024 at 7:48 PM Laurent Goujon >> > > > <laur...@dremio.com.invalid> wrote: >> > > > > >> > > > > Hello Arrow Java developers, >> > > > > >> > > > > I would wonder if the community would be okay to change the >> minimum >> > Java >> > > > > version used by the build toolchain to at least Java 17 or 21 (or >> > even >> > > > 22). >> > > > > This is different from changing the minimum Java version used at >> > runtime >> > > > > which would still be 8 (following the vote from last september). >> > > > > >> > > > > Concretely it would mean: >> > > > > * Java 21 would be required to build Arrow Java >> > > > > * But Arrow would still be compatible with Java 8 >> > > > > * Unit tests should keep running with Java 8 and higher. >> > > > > >> > > > > Reasons for changing the toolchain would be: >> > > > > - More and more tools and plugins now require at least Java 11, >> > forcing >> > > > the >> > > > > project to keep using older/unsupported versions >> > > > > - We hacked our way to support Java modules with >> > > > > >> > > > >> > >> https://github.com/apache/arrow/tree/main/java/maven/module-info-compiler-maven-plugin >> > > > > - There are several new features which we could conditionally >> > include in >> > > > > the Arrow project like VarHandle (Java 9) and Foreign Function and >> > Memory >> > > > > API (Java 22) to move away from Unsafe support which require more >> and >> > > > more >> > > > > workarounds (Apache Lucene is a project which has managed to >> > introduce >> > > > > support for multiple Java incubator and final API while >> maintaining >> > > > > compatibility with previous Java versions). But doing it from >> Java 8 >> > > > > creates a higher barrier. >> > > > > >> > > > > Laurent >> > > > >> > >> >
