Sure thing, I will send a PR. Curtis
On Tue, Mar 12, 2024, 11:32 AM Keyong Zhou <zho...@apache.org> wrote: > I think so. Do you mind sending a PR to Celeborn? :) > > Regards, > Keyong Zhou > > Curtis Howard <curtis.james.how...@gmail.com>于2024年3月12日 周二22:58写道： > >> Thanks Keyong, >> >> I was able to build the 'client' JARs for Spark 3.2 / JDK21, and run >> simple tests; however I did face a runtime error similar to this: >> >> Caused by: java.lang.ExceptionInInitializerError: Exception >> java.lang.IllegalStateException: java.lang.NoSuchMethodException: >> java.nio.DirectByteBuffer.<init>(long, int) [in thread "Executor task >> launch worker for task 0.0 in stage 0.0 (TID 0)"] >> >> at >> org.apache.celeborn.common.unsafe.Platform.<clinit>(Platform.java:135) >> >> ... 16 more >> >> I used the following patch (borrowed from the almost identical upstream >> Spark project patch for SPARK-42369 >> <https://github.com/apache/spark/pull/39909> that was required for >> JDK21, as a result of JDK-8303083 >> <https://bugs.openjdk.org/browse/JDK-8303083>), to work around this >> successfully (I think it may be required in Celeborn Platform.java as well, >> for JDK21?): >> >> diff --git >> a/common/src/main/java/org/apache/celeborn/common/unsafe/Platform.java >> b/common/src/main/java/org/apache/celeborn/common/unsafe/Platform.java >> >> index ec541a77..218d517b 100644 >> >> --- a/common/src/main/java/org/apache/celeborn/common/unsafe/Platform.java >> >> +++ b/common/src/main/java/org/apache/celeborn/common/unsafe/Platform.java >> >> @ -90,7 +90,15 @@ public final class Platform { >> >> } >> >> try { >> >> Class<?> cls = Class.forName("java.nio.DirectByteBuffer"); >> >> - Constructor<?> constructor = >> cls.getDeclaredConstructor(Long.TYPE, Integer.TYPE); >> >> + Constructor<?> constructor; >> >> + try { >> >> + constructor = cls.getDeclaredConstructor(Long.TYPE, >> Integer.TYPE); >> >> + } catch (NoSuchMethodException e) { >> >> + // DirectByteBuffer(long,int) was removed in >> >> + // >> https://github.com/openjdk/jdk/commit/a56598f5a534cc9223367e7faa8433ea38661db9 >> >> + constructor = cls.getDeclaredConstructor(Long.TYPE, Long.TYPE); >> >> + } >> >> + >> >> Field cleanerField = cls.getDeclaredField("cleaner"); >> >> try { >> >> constructor.setAccessible(true); >> >> >> >> >> Curtis >> >> >> >> On Tue, Mar 12, 2024 at 10:32 AM Keyong Zhou <zho...@apache.org> wrote: >> >>> Hi Curtis, >>> >>> With this PR https://github.com/apache/incubator-celeborn/pull/2385 you >>> can compile with JDK21 using following commands: >>> ./build/make-distribution.sh -Pspark-3.5 -Pjdk-21 >>> >>> Regards, >>> Keyong Zhou >>> >>> Curtis Howard <curtis.james.how...@gmail.com> 于2024年3月9日周六 02:38写道： >>> >>>> Thank you Keyong! >>>> >>>> Related to this, has testing started for Celeborn with JDK21? (any >>>> anticipated concerns there, based on what you know so far?). >>>> We will be migrating to JDK21 shortly, which is why I ask. >>>> >>>> Thanks again >>>> Curtis >>>> >>>> On Fri, Mar 8, 2024 at 11:05 AM Keyong Zhou <zho...@apache.org> wrote: >>>> >>>>> Hi Curtis, >>>>> >>>>> Thanks for reaching out! >>>>> >>>>> No there is no known blockers for Celeborn + Spark 3.2 + JDK17, and I >>>>> think there is a good chance that it could be used successfully. >>>>> >>>>> Any problems with your test, feel free to let us know :) >>>>> >>>>> Regards, >>>>> Keyong Zhou >>>>> >>>>> Curtis Howard <curtis.james.how...@gmail.com> 于2024年3月8日周五 22:23写道： >>>>> >>>>>> Hi, >>>>>> >>>>>> We would like to confirm the reason for Celeborn not being listed >>>>>> with Spark 3.2 and JDK17, as shown in the compatibility matrix here: >>>>>> https://github.com/apache/incubator-celeborn?tab=readme-ov-file#build >>>>>> >>>>>> Is the reason for this only because the Apache Spark 3.2 release does >>>>>> not officially support JDK17? (as covered in >>>>>> https://issues.apache.org/jira/browse/SPARK-33772), or have >>>>>> another Celeborn-specific conflicts been found, with the Spark 3.2 + >>>>>> JDK17 >>>>>> version combination? >>>>>> >>>>>> We currently build Spark ourselves with custom dependencies, and are >>>>>> successfully using Spark 3.2 with JDK17. Understanding that this >>>>>> combination has likely not been tested with Celeborn, we are wondering if >>>>>> there are any known blockers for Celeborn + Spark 3.2 + JDK17, or if >>>>>> there >>>>>> is a good chance that it could still be used successfully. >>>>>> >>>>>> Thank you! >>>>>> Curtis >>>>>> >>>>>
