Thanks all for the input. It sounds like it is NOT feasible to maintain a legacy branch. So when Arrow drops Java 8 (and 11?) support, we should consider it EOL from Arrow's perspective.
+1 to Fokko's suggestion on waiting. I propose we commit to dropping Java 8 and 11 from Arrow in tandem with the Spark 4 release. Spark 4 has a release date of 2024-06[1]. I also propose that we re-discuss closer to that date as a final evaluation. This was also discussed in the Arrow bi-weekly sync where waiting for Spark 4 was agreed upon. There is a lot of pre-work we can do to prepare Arrow Java for new features that will become available. [1]https://lists.apache.org/thread/xhkgj60j361gdpywoxxz7qspp2w80ry6 -Dane On Wed, Oct 11, 2023 at 2:50 AM Fokko Driesprong <fo...@apache.org> wrote: > Hey everyone, > > Great to bring this up. I can speak for the Iceberg community, and we > expect to support Java 8 for a long time there (unfortunately). Let me go > over some of the arguments here. > > Java 8 does have a long Extended Support timeline, but a recent > > report shows Java 11 increasing in adoption vs Java 8. "More than 56% of > > applications are now using Java 11 in production (up from 48% in 2022 and > > 11% in 2020). Java 8 is a close second with nearly 33% of applications > > using it in production (down from 46% in 2022)."[2] > > > I think this is skewed, mostly because it is easy to upgrade a Spring > application to the latest version from Java, but if you're tied to the > Hadoop ecosystems, things are moving slowly. > > 2. Unblock Arrow from upgrading dependencies that no longer support Java 8. > > > Following the links, I noticed the dependencies are around tests (Mockito) > and static error checking (error-prone). Those sound like nice to have to > me. > > An example; Apache Thrift dropped Java 8 a while ago but added the support > again since it was breaking the ecosystem > <https://github.com/apache/thrift/pull/2785>. Thrift is used in Parquet, > and in Parquet we cannot yet drop Java 8 support. I think of Arrow as a > low-level library, like Thirft and Iceberg, and I think it makes sense to > serve an as wide audience as possible (within reasonable bounds of course). > > I would at least wait until the release of Spark 4. My experience is that > nobody is really eager to do backporting to older versions, and for me, I > still think the gains of dropping Java 8 support are not that big. > > Kind regards, > Fokko Driesprong > > Op wo 11 okt 2023 om 06:05 schreef Jacob Wujciak-Jens > <ja...@voltrondata.com.invalid>: > > > > I cannot estimate the effort to backport large features like the new > > layouts that are currently being added (e.g. RunEndEncoding, ListView, > > etc.). > > > > In my mind we are only talking about patch releases for security fixes or > > similarly critical issues as otherwise the effort to maintain 'v14' (but > > actually arrow-latest) would surely overshadow any gains made by > > deprecating jdk 8? > > > > On Wed, Oct 11, 2023 at 3:31 AM Gang Wu <ust...@gmail.com> wrote: > > > > > I agree that we have to move on. It seems that patch release to > > > Arrow v14 is a good idea, though I cannot estimate the effort to > > > backport large features like the new layouts that are currently > > > being added (e.g. RunEndEncoding, ListView, etc.). > > > > > > As an Arrow developer, I am always happy to drop JDK 8. My > > > employer has leveraged Apache Arrow in the internal engine > > > and depends on Arrow Java in the Java SDK. For end users > > > who cannot get away with JDK 8, we might need to prepare > > > different Java SDKs and use features that are available in the > > > Arrow v14, or let the server side chooses which subset of > > > features based on the SDK version. > > > > > > Thanks, > > > Gang > > > > > > > > > > > > On Wed, Oct 11, 2023 at 12:40 AM Dane Pitkin > > <d...@voltrondata.com.invalid > > > > > > > wrote: > > > > > > > To summarize the discussion so far: > > > > > > > > * Some Arrow Java users are still on JDK 8 > > > > * Arrow v14 is proposed as the final version with JDK 8 support > > > > * Arrow v14 can support patch releases if necessary for JDK 8 users > > > > * There is an open question to decide if JDK 11 should be dropped > > > > simultaneously > > > > > > > > Gang Wu, I'm curious what are your thoughts given your initial > > concerns? > > > > > > > > -Dane > > > > > > > > On Sat, Oct 7, 2023 at 12:00 AM Jacob Wujciak-Jens > > > > <ja...@voltrondata.com.invalid> wrote: > > > > > > > > > From a release engineer perspective (without java knowledge) I > agree > > > with > > > > > Micah, I'd rather make a patch release for an older version if > needed > > > but > > > > > modernize the codebase and simplify CI! > > > > > > > > > > > > > > > On Sat, Oct 7, 2023 at 5:27 AM Micah Kornfield < > > emkornfi...@gmail.com> > > > > > wrote: > > > > > > > > > > > I think given the stability of Arrow Java, dropping support > > probably > > > > > makes > > > > > > sense. If a bug comes up or consumers really need to new > features > > we > > > > can > > > > > > always make a patch release of an older version. > > > > > > > > > > > > On Thu, Oct 5, 2023 at 3:13 PM Dane Pitkin > > > > <d...@voltrondata.com.invalid > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I also learned today that Apache Spark has dropped support for > > > Java 8 > > > > > and > > > > > > > 11 for their next release (v4.0)[1]. Should we consider > dropping > > > Java > > > > > 11 > > > > > > as > > > > > > > well? > > > > > > > > > > > > > > [1]https://github.com/apache/spark/pull/43005 > > > > > > > > > > > > > > -Dane > > > > > > > > > > > > > > On Thu, Oct 5, 2023 at 3:30 PM Dane Pitkin < > d...@voltrondata.com > > > > > > > > wrote: > > > > > > > > > > > > > > > I created a GH issue[1] proposing the removal of Java 8 > > support. > > > It > > > > > > > > would target the Arrow v15 release (~Jan 2024). > > > > > > > > > > > > > > > > IMO it would be in the best interest of the project for two > > major > > > > > > > reasons: > > > > > > > > 1. Unblock the Java Platform Module System (JPMS)[2] > > > > implementation. > > > > > > > > 2. Unblock Arrow from upgrading dependencies that no longer > > > support > > > > > > Java > > > > > > > > 8. (See [1] for examples) > > > > > > > > > > > > > > > > Since Arrow Java has been quite stable, will Java 8 users be > > okay > > > > > with > > > > > > > > pinning Arrow to the last supported release (v14) if the > Arrow > > > > > project > > > > > > > > ultimately decides to remove Java 8 support? > > > > > > > > > > > > > > > > > > > > > > > > [1]https://github.com/apache/arrow/issues/38051 > > > > > > > > [2]https://en.wikipedia.org/wiki/Java_Platform_Module_System > > > > > > > > > > > > > > > > -Dane > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 12:26 PM Dane Pitkin < > > > d...@voltrondata.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> - As a low level library, users have to add specific flags > to > > > use > > > > > > > >>> Java 9 and up with Arrow to resolve issues with java.nio. > > This > > > > has > > > > > > > >>> been annoying for our customers constantly. If this is not > > > > > resolved, > > > > > > > >>> I would say we may see a lot of complaints in the future. > > > > > > > >>> > > > > > > > >> I filed issue 37739[1] to track this, but it sounds like > this > > > > can't > > > > > be > > > > > > > >> changed until Java 21 or 24. > > > > > > > >> > > > > > > > >> - It seems that the EOL of Java 8 from Oracle is Dec 2030 > > [2]. A > > > > lot > > > > > > > >>> users will still stay on it for a long time. At least this > > is > > > > true > > > > > > for > > > > > > > >>> our > > > > > > > >>> customers. So I am afraid we may not upgrade to newer > > versions > > > > > > > >>> of Arrow if it no longer supports Java 8. > > > > > > > >>> > > > > > > > >> Java 8 does have a long Extended Support timeline, but a > > recent > > > > > > > >> report shows Java 11 increasing in adoption vs Java 8. "More > > > than > > > > > 56% > > > > > > of > > > > > > > >> applications are now using Java 11 in production (up from > 48% > > in > > > > > 2022 > > > > > > > and > > > > > > > >> 11% in 2020). Java 8 is a close second with nearly 33% of > > > > > applications > > > > > > > >> using it in production (down from 46% in 2022)."[2] > > > > > > > >> I expect the Java ecosystem will find a way to move on from > > > Java 8 > > > > > > much > > > > > > > >> sooner than 2030, meaning many of Arrow's dependencies could > > > drop > > > > > > > support > > > > > > > >> for Java 8 before then. At this point, Arrow may be forced > to > > > > > support > > > > > > a > > > > > > > >> higher minimum Java version. > > > > > > > >> > > > > > > > >> That being said, it's hard to argue against real use cases. > > I'd > > > be > > > > > > > >> curious to hear what Java version other users of Arrow are > > using > > > > > (and > > > > > > if > > > > > > > >> there is a timeline to upgrade if on Java 8). > > > > > > > >> > > > > > > > >> > > > > > > > >> [1]https://github.com/apache/arrow/issues/37739 > > > > > > > >> [2] > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://newrelic.com/sites/default/files/2023-04/new-relic-2023-state-of-the-java-ecosystem-2023-04-20.pdf > > > > > > > >> > > > > > > > >> > > > > > > > >> -Dane > > > > > > > >> > > > > > > > >> > > > > > > > >> On Thu, Sep 14, 2023 at 11:45 AM Gang Wu <ust...@gmail.com> > > > > wrote: > > > > > > > >> > > > > > > > >>> Thanks for bringing this up! > > > > > > > >>> > > > > > > > >>> I have two concerns of dropping Java 8 support: > > > > > > > >>> - As a low level library, users have to add specific flags > > [1] > > > to > > > > > use > > > > > > > >>> Java 9 and up with Arrow to resolve issues with java.nio. > > This > > > > has > > > > > > > >>> been annoying for our customers constantly. If this is not > > > > > resolved, > > > > > > > >>> I would say we may see a lot of complaints in the future. > > > > > > > >>> - It seems that the EOL of Java 8 from Oracle is Dec 2030 > > [2]. > > > A > > > > > lot > > > > > > > >>> users will still stay on it for a long time. At least this > > is > > > > true > > > > > > for > > > > > > > >>> our > > > > > > > >>> customers. So I am afraid we may not upgrade to newer > > versions > > > > > > > >>> of Arrow if it no longer supports Java 8. > > > > > > > >>> > > > > > > > >>> [1] > > > > > > > https://arrow.apache.org/docs/java/install.html#java-compatibility > > > > > > > >>> [2] > > > > > > > >>> > > > > > > > > > https://www.oracle.com/java/technologies/java-se-support-roadmap.html > > > > > > > >>> > > > > > > > >>> Best, > > > > > > > >>> Gang > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> On Thu, Sep 14, 2023 at 11:14 PM David Dali Susanibar Arce > < > > > > > > > >>> davi.sar...@gmail.com> wrote: > > > > > > > >>> > > > > > > > >>> > Hi Arrow Java developers, > > > > > > > >>> > > > > > > > > >>> > I would like to propose a timeline for dropping support > for > > > > Java > > > > > 8: > > > > > > > >>> > - Propose to drop JDK8 in Arrow v15 (2 releases from now) > > > > > > > >>> > - JDK 21 support will be added before removal of JDK8 > > > > > > > >>> > > > > > > > > >>> > Why? > > > > > > > >>> > - Java 8 no longer receives Premier Support (1) > > > > > > > >>> > - Some Arrow Java (test) dependencies have already > started > > to > > > > > drop > > > > > > > >>> > Java 8 support, forcing us to pin to older packager > > versions > > > > > > > >>> > > > > > > > > >>> > Also note: > > > > > > > >>> > - gRPC Java may drop support for a JDK version when that > > > > version > > > > > is > > > > > > > no > > > > > > > >>> > longer receiving Premier Support from Oracle (2), more > > detail > > > > at > > > > > > Java > > > > > > > >>> > 8 / Java 11 support timeline in gRPC here (3) > > > > > > > >>> > - Spark plans to tentatively drop JDK 8 support in Spark > > 4.0 > > > > (4), > > > > > > > >>> > which has a release timeline of approximately 2024-06 > (5). > > Is > > > > it > > > > > > fine > > > > > > > >>> > for us to drop JDK 8 support before spark? > > > > > > > >>> > > > > > > > > >>> > (1) > > > > > > > >>> > > > > > > > > > https://www.oracle.com/java/technologies/java-se-support-roadmap.html > > > > > > > >>> > (2) > > > > > > > >>> > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/grpc/proposal/pull/283/files#:~:text=gRPC%20Java%20may,support%5D > > > > > > > >>> > . > > > > > > > >>> > (3) > > > > > > https://groups.google.com/g/grpc-io/c/-XK6Kd_19YQ/m/-4s07TzdAgAJ > > > > > > > >>> > (4) https://issues.apache.org/jira/browse/SPARK-44112 > > > > > > > >>> > (5) > > > > > > https://www.mail-archive.com/dev@spark.apache.org/msg30460.html > > > > > > > >>> > > > > > > > > >>> > Consider: > > > > > > > >>> > - JDK8 deprecation is currently not mandatory. We simply > > want > > > > to > > > > > > > >>> > devote more time to development of Java LTS versions 11, > 17 > > > and > > > > > 21. > > > > > > > >>> > - Java 11 is dropping Premier Support this month. > > > > > > > >>> > > > > > > > > >>> > Best regards, > > > > > > > >>> > > > > > > > > >>> > -- > > > > > > > >>> > David Susanibar > > > > > > > >>> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
