unsubscribe

[Ravi Chinoy]

-- 
Regards
Ravi Chinoy
Phone: (415) 230 9971

RE: Spark on Yarn with Java 17

[Luca Canali]

Jason, In case you need a pointer on how to run Spark with a version of Java 
different than the version used by the Hadoop processes, as indicated by 
Dongjoon, this is an example of what we do on our Hadoop clusters: 
https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_Set_Java_Home_Howto.md

Best,
Luca

From: Dongjoon Hyun 
Sent: Saturday, December 9, 2023 09:39
To: Jason Xu 
Cc: dev@spark.apache.org
Subject: Re: Spark on Yarn with Java 17

Please try Apache Spark 3.3+ (SPARK-33772) with Java 17 on your cluster simply, 
Jason.

I believe you can set up for your Spark 3.3+ jobs to run with Java 17 while 
your cluster(DataNode/NameNode/ResourceManager/NodeManager) is still sitting on 
Java 8.

Dongjoon.

On Fri, Dec 8, 2023 at 11:12 PM Jason Xu 
mailto:jasonxu.sp...@gmail.com>> wrote:
Dongjoon, thank you for the fast response!

Apache Spark 4.0.0 depends on only Apache Hadoop client library.

To better understand your answer, does that mean a Spark application built with 
Java 17 can successfully run on a Hadoop cluster on version 3.3 and Java 8 
runtime?

On Fri, Dec 8, 2023 at 4:33 PM Dongjoon Hyun 
mailto:dongj...@apache.org>> wrote:
Hi, Jason.

Apache Spark 4.0.0 depends on only Apache Hadoop client library.

You can track all `Apache Spark 4` activities including Hadoop dependency here.

https://issues.apache.org/jira/browse/SPARK-44111
(Prepare Apache Spark 4.0.0)

According to the release history, the original suggested timeline was June, 
2024.
- Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
- Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
- Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
- Spark 4: 2024.06 (4.0.0, NEW)

Thanks,
Dongjoon.

On 2023/12/08 23:50:15 Jason Xu wrote:
> Hi Spark devs,
>
> According to the Spark 3.5 release notes, Spark 4 will no longer support
> Java 8 and 11 (link
> 
> ).
>
> My company is using Spark on Yarn with Java 8 now. When considering a
> future upgrade to Spark 4, one issue we face is that the latest version of
> Hadoop (3.3) does not yet support Java 17. There is an open ticket (
> HADOOP-17177 ) for this
> issue, which has been open for over two years.
>
> My question is: Does the release of Spark 4 depend on the availability of
> Java 17 support in Hadoop? Additionally, do we have a rough estimate for
> the release of Spark 4? Thanks!
>
>
> Cheers,
>
> Jason Xu
>

-
To unsubscribe e-mail: 
dev-unsubscr...@spark.apache.org

Re: Spark on Yarn with Java 17

[Dongjoon Hyun]

Please try Apache Spark 3.3+ (SPARK-33772) with Java 17 on your cluster
simply, Jason.

I believe you can set up for your Spark 3.3+ jobs to run with Java 17 while
your cluster(DataNode/NameNode/ResourceManager/NodeManager) is still
sitting on Java 8.

Dongjoon.

On Fri, Dec 8, 2023 at 11:12 PM Jason Xu  wrote:

> Dongjoon, thank you for the fast response!
>
> Apache Spark 4.0.0 depends on only Apache Hadoop client library.
>
> To better understand your answer, does that mean a Spark application built
> with Java 17 can successfully run on a Hadoop cluster on version 3.3 and
> Java 8 runtime?
>
> On Fri, Dec 8, 2023 at 4:33 PM Dongjoon Hyun  wrote:
>
>> Hi, Jason.
>>
>> Apache Spark 4.0.0 depends on only Apache Hadoop client library.
>>
>> You can track all `Apache Spark 4` activities including Hadoop dependency
>> here.
>>
>> https://issues.apache.org/jira/browse/SPARK-44111
>> (Prepare Apache Spark 4.0.0)
>>
>> According to the release history, the original suggested timeline was
>> June, 2024.
>> - Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
>> - Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
>> - Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
>> - Spark 4: 2024.06 (4.0.0, NEW)
>>
>> Thanks,
>> Dongjoon.
>>
>> On 2023/12/08 23:50:15 Jason Xu wrote:
>> > Hi Spark devs,
>> >
>> > According to the Spark 3.5 release notes, Spark 4 will no longer support
>> > Java 8 and 11 (link
>> > <
>> https://spark.apache.org/releases/spark-release-3-5-0.html#upcoming-removal
>> >
>> > ).
>> >
>> > My company is using Spark on Yarn with Java 8 now. When considering a
>> > future upgrade to Spark 4, one issue we face is that the latest version
>> of
>> > Hadoop (3.3) does not yet support Java 17. There is an open ticket (
>> > HADOOP-17177 ) for
>> this
>> > issue, which has been open for over two years.
>> >
>> > My question is: Does the release of Spark 4 depend on the availability
>> of
>> > Java 17 support in Hadoop? Additionally, do we have a rough estimate for
>> > the release of Spark 4? Thanks!
>> >
>> >
>> > Cheers,
>> >
>> > Jason Xu
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>