[DISCUSSION]JDK11 for Apache 2.x?

[DB Tsai]

Hello everyone,

Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a 
community.

Java 8 is already end of life for commercial users, and many companies are 
moving to Java 11. 
The release date for Apache Spark 3.0 is still not there yet, and there are 
many API 
incompatibility issues when upgrading from Spark 2.x. As a result, asking users 
to move to
Spark 3.0 to use JDK 11 is not realistic.

Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?

Should we cut a new Apache Spark 2.5 since the patches involve some of the 
dependencies changes
which is not desired in minor release?

Thanks.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSSION]JDK11 for Apache 2.x?

[Holden Karau]

On Tue, Aug 27, 2019 at 12:01 PM DB Tsai  wrote:

> Hello everyone,
>
> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a
> community.
>
> Java 8 is already end of life for commercial users, and many companies are
> moving to Java 11.
> The release date for Apache Spark 3.0 is still not there yet, and there
> are many API
> incompatibility issues when upgrading from Spark 2.x. As a result, asking
> users to move to
> Spark 3.0 to use JDK 11 is not realistic.
>
> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
>
> Should we cut a new Apache Spark 2.5 since the patches involve some of the
> dependencies changes
> which is not desired in minor release?

I think a LTS 2.5 branch with JDK 11 would make sense given the state of
Java 8.
I think backporting to 2.4 would be not ideal.

>
>
> Thanks.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [DISCUSSION]JDK11 for Apache 2.x?

[Sean Owen]

I think one of the key problems here are the required dependency
upgrades. It would mean many minor breaking changes and a few bigger
ones, notably around Hive, and forces a scala 2.12-only update. I
think my question is whether that even makes sense as a minor release?
it wouldn't be backwards compatible with 2.4 enough to call it a
low-risk update. It would be a smaller step than moving all the way to
3.0, sure. I am not super against it, but we have to keep in mind how
much work it would then be to maintain two LTS 2.x releases, 2.4 and
the sort-of-compatible 2.5, while proceeding with 3.x.

On Tue, Aug 27, 2019 at 2:01 PM DB Tsai  wrote:
>
> Hello everyone,
>
> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a 
> community.
>
> Java 8 is already end of life for commercial users, and many companies are 
> moving to Java 11.
> The release date for Apache Spark 3.0 is still not there yet, and there are 
> many API
> incompatibility issues when upgrading from Spark 2.x. As a result, asking 
> users to move to
> Spark 3.0 to use JDK 11 is not realistic.
>
> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
>
> Should we cut a new Apache Spark 2.5 since the patches involve some of the 
> dependencies changes
> which is not desired in minor release?
>
> Thanks.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
> Inc
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSSION]JDK11 for Apache 2.x?

[Jean Georges Perrin]

Not a contributor, but a user perspective…

As Spark 3.x will be an evolution, I am not completely shocked that it would 
imply a Java 11 requirement as well. Would be great to have both Java 8 and 
Java 11, but one needs to be able to say goodbye. Java 8 is great, still using 
it actively in production, but we know its time is limited, so, by the time we 
evolve to Spark 3, we could combine it with Java 11.

On the other hand, not everybody may think this way and it may slow down the 
adoption of Spark 3…

However, I concur with Sean, I don’t think another 2.x is needed for Java 11.

> On Aug 27, 2019, at 3:09 PM, Sean Owen  wrote:
> 
> I think one of the key problems here are the required dependency
> upgrades. It would mean many minor breaking changes and a few bigger
> ones, notably around Hive, and forces a scala 2.12-only update. I
> think my question is whether that even makes sense as a minor release?
> it wouldn't be backwards compatible with 2.4 enough to call it a
> low-risk update. It would be a smaller step than moving all the way to
> 3.0, sure. I am not super against it, but we have to keep in mind how
> much work it would then be to maintain two LTS 2.x releases, 2.4 and
> the sort-of-compatible 2.5, while proceeding with 3.x.
> 
> On Tue, Aug 27, 2019 at 2:01 PM DB Tsai  wrote:
>> 
>> Hello everyone,
>> 
>> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a 
>> community.
>> 
>> Java 8 is already end of life for commercial users, and many companies are 
>> moving to Java 11.
>> The release date for Apache Spark 3.0 is still not there yet, and there are 
>> many API
>> incompatibility issues when upgrading from Spark 2.x. As a result, asking 
>> users to move to
>> Spark 3.0 to use JDK 11 is not realistic.
>> 
>> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
>> 
>> Should we cut a new Apache Spark 2.5 since the patches involve some of the 
>> dependencies changes
>> which is not desired in minor release?
>> 
>> Thanks.
>> 
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, 
>> Inc
>> 
>> 
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> 
> 
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> 


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSSION]JDK11 for Apache 2.x?

[Sean Owen]

Spark 3 will not require Java 11; it will work with Java 8 too. I
think the question is whether someone who _wants_ Java 11 should have
a 2.x release instead of 3.0.

In practice... how much are people moving off Java 8 right now? It's
still my default, because most but not all things work with 11. (Here,
I'm using 11 as shorthand for 9+; it's the LTS release in the 9 to 11
series).
I get that the issue is support, but it seems that Oracle is still
providing public updates through Dec 2020. I think it's that
commercial support is now for-pay? And OpenJDK 8 still gets updates?
that I'm unclear on.

Not that those aren't legitimate concerns, I'm just trying to figure
out how much need vs want there is and why out there.


On Tue, Aug 27, 2019 at 4:15 PM Jean Georges Perrin  wrote:
>
> Not a contributor, but a user perspective…
>
> As Spark 3.x will be an evolution, I am not completely shocked that it would 
> imply a Java 11 requirement as well. Would be great to have both Java 8 and 
> Java 11, but one needs to be able to say goodbye. Java 8 is great, still 
> using it actively in production, but we know its time is limited, so, by the 
> time we evolve to Spark 3, we could combine it with Java 11.
>
> On the other hand, not everybody may think this way and it may slow down the 
> adoption of Spark 3…
>
> However, I concur with Sean, I don’t think another 2.x is needed for Java 11.
>
> > On Aug 27, 2019, at 3:09 PM, Sean Owen  wrote:
> >
> > I think one of the key problems here are the required dependency
> > upgrades. It would mean many minor breaking changes and a few bigger
> > ones, notably around Hive, and forces a scala 2.12-only update. I
> > think my question is whether that even makes sense as a minor release?
> > it wouldn't be backwards compatible with 2.4 enough to call it a
> > low-risk update. It would be a smaller step than moving all the way to
> > 3.0, sure. I am not super against it, but we have to keep in mind how
> > much work it would then be to maintain two LTS 2.x releases, 2.4 and
> > the sort-of-compatible 2.5, while proceeding with 3.x.
> >
> > On Tue, Aug 27, 2019 at 2:01 PM DB Tsai  wrote:
> >>
> >> Hello everyone,
> >>
> >> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a 
> >> community.
> >>
> >> Java 8 is already end of life for commercial users, and many companies are 
> >> moving to Java 11.
> >> The release date for Apache Spark 3.0 is still not there yet, and there 
> >> are many API
> >> incompatibility issues when upgrading from Spark 2.x. As a result, asking 
> >> users to move to
> >> Spark 3.0 to use JDK 11 is not realistic.
> >>
> >> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
> >>
> >> Should we cut a new Apache Spark 2.5 since the patches involve some of the 
> >> dependencies changes
> >> which is not desired in minor release?
> >>
> >> Thanks.
> >>
> >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   
> >> Apple, Inc
> >>
> >>
> >> -
> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSSION]JDK11 for Apache 2.x?

[Holden Karau]

So I'm not a lawyer, but looking at the JDK8 License FAQ (
https://www.oracle.com/technetwork/java/javase/overview/oracle-jdk-faqs.html )
makes me feel like we should support folks who are on the 2.x branch and
want to migrate away from JDK8. Although the Redhat post about OpenJDK8 is
reassuring (
https://developers.redhat.com/blog/2019/05/22/openjdk-8-and-11-still-in-safe-hands/
 ).

On Tue, Aug 27, 2019 at 2:22 PM Sean Owen  wrote:

> Spark 3 will not require Java 11; it will work with Java 8 too. I
> think the question is whether someone who _wants_ Java 11 should have
> a 2.x release instead of 3.0.
>
> In practice... how much are people moving off Java 8 right now? It's
> still my default, because most but not all things work with 11. (Here,
> I'm using 11 as shorthand for 9+; it's the LTS release in the 9 to 11
> series).
> I get that the issue is support, but it seems that Oracle is still
> providing public updates through Dec 2020. I think it's that
> commercial support is now for-pay? And OpenJDK 8 still gets updates?
> that I'm unclear on.
>
> Not that those aren't legitimate concerns, I'm just trying to figure
> out how much need vs want there is and why out there.
>
>
> On Tue, Aug 27, 2019 at 4:15 PM Jean Georges Perrin  wrote:
> >
> > Not a contributor, but a user perspective…
> >
> > As Spark 3.x will be an evolution, I am not completely shocked that it
> would imply a Java 11 requirement as well. Would be great to have both Java
> 8 and Java 11, but one needs to be able to say goodbye. Java 8 is great,
> still using it actively in production, but we know its time is limited, so,
> by the time we evolve to Spark 3, we could combine it with Java 11.
> >
> > On the other hand, not everybody may think this way and it may slow down
> the adoption of Spark 3…
> >
> > However, I concur with Sean, I don’t think another 2.x is needed for
> Java 11.
> >
> > > On Aug 27, 2019, at 3:09 PM, Sean Owen  wrote:
> > >
> > > I think one of the key problems here are the required dependency
> > > upgrades. It would mean many minor breaking changes and a few bigger
> > > ones, notably around Hive, and forces a scala 2.12-only update. I
> > > think my question is whether that even makes sense as a minor release?
> > > it wouldn't be backwards compatible with 2.4 enough to call it a
> > > low-risk update. It would be a smaller step than moving all the way to
> > > 3.0, sure. I am not super against it, but we have to keep in mind how
> > > much work it would then be to maintain two LTS 2.x releases, 2.4 and
> > > the sort-of-compatible 2.5, while proceeding with 3.x.
> > >
> > > On Tue, Aug 27, 2019 at 2:01 PM DB Tsai 
> wrote:
> > >>
> > >> Hello everyone,
> > >>
> > >> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as
> a community.
> > >>
> > >> Java 8 is already end of life for commercial users, and many
> companies are moving to Java 11.
> > >> The release date for Apache Spark 3.0 is still not there yet, and
> there are many API
> > >> incompatibility issues when upgrading from Spark 2.x. As a result,
> asking users to move to
> > >> Spark 3.0 to use JDK 11 is not realistic.
> > >>
> > >> Should we backport PRs for JDK11 and cut a release in 2.x to support
> JDK11?
> > >>
> > >> Should we cut a new Apache Spark 2.5 since the patches involve some
> of the dependencies changes
> > >> which is not desired in minor release?
> > >>
> > >> Thanks.
> > >>
> > >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |  
> Apple, Inc
> > >>
> > >>
> > >> -
> > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >>
> > >
> > > -
> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >
> >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [DISCUSSION]JDK11 for Apache 2.x?

[Sean Owen]

That's just the Oracle 'commercial' JDK though. OpenJDK is still OSS
and 8 is updated there too. Oracle JDK is still free too, just not for
commercial use and commercial support for 8 (IIUC). I think that's the
'audience' here, and I don't know how large or small that is.

That said a key first question is simply, how much breaking change is
necessary to get 2.x to work on JDK 11? If it's large I think that
might resolve the question right there. SPARK-24417 has most but not
necessarily all necessary changes under it.

Heck it could be maintained as a separate community project too, if
there is both interest from some parties to make it happen, but not
enough to commit to maintain all the backports and updated from 3.x.

On Sun, Sep 1, 2019 at 9:04 PM Holden Karau  wrote:
>
> So I'm not a lawyer, but looking at the JDK8 License FAQ ( 
> https://www.oracle.com/technetwork/java/javase/overview/oracle-jdk-faqs.html 
> ) makes me feel like we should support folks who are on the 2.x branch and 
> want to migrate away from JDK8. Although the Redhat post about OpenJDK8 is 
> reassuring ( 
> https://developers.redhat.com/blog/2019/05/22/openjdk-8-and-11-still-in-safe-hands/
>  ).
>
> On Tue, Aug 27, 2019 at 2:22 PM Sean Owen  wrote:
>>
>> Spark 3 will not require Java 11; it will work with Java 8 too. I
>> think the question is whether someone who _wants_ Java 11 should have
>> a 2.x release instead of 3.0.
>>
>> In practice... how much are people moving off Java 8 right now? It's
>> still my default, because most but not all things work with 11. (Here,
>> I'm using 11 as shorthand for 9+; it's the LTS release in the 9 to 11
>> series).
>> I get that the issue is support, but it seems that Oracle is still
>> providing public updates through Dec 2020. I think it's that
>> commercial support is now for-pay? And OpenJDK 8 still gets updates?
>> that I'm unclear on.
>>
>> Not that those aren't legitimate concerns, I'm just trying to figure
>> out how much need vs want there is and why out there.
>>
>>
>> On Tue, Aug 27, 2019 at 4:15 PM Jean Georges Perrin  wrote:
>> >
>> > Not a contributor, but a user perspective…
>> >
>> > As Spark 3.x will be an evolution, I am not completely shocked that it 
>> > would imply a Java 11 requirement as well. Would be great to have both 
>> > Java 8 and Java 11, but one needs to be able to say goodbye. Java 8 is 
>> > great, still using it actively in production, but we know its time is 
>> > limited, so, by the time we evolve to Spark 3, we could combine it with 
>> > Java 11.
>> >
>> > On the other hand, not everybody may think this way and it may slow down 
>> > the adoption of Spark 3…
>> >
>> > However, I concur with Sean, I don’t think another 2.x is needed for Java 
>> > 11.
>> >
>> > > On Aug 27, 2019, at 3:09 PM, Sean Owen  wrote:
>> > >
>> > > I think one of the key problems here are the required dependency
>> > > upgrades. It would mean many minor breaking changes and a few bigger
>> > > ones, notably around Hive, and forces a scala 2.12-only update. I
>> > > think my question is whether that even makes sense as a minor release?
>> > > it wouldn't be backwards compatible with 2.4 enough to call it a
>> > > low-risk update. It would be a smaller step than moving all the way to
>> > > 3.0, sure. I am not super against it, but we have to keep in mind how
>> > > much work it would then be to maintain two LTS 2.x releases, 2.4 and
>> > > the sort-of-compatible 2.5, while proceeding with 3.x.
>> > >
>> > > On Tue, Aug 27, 2019 at 2:01 PM DB Tsai  wrote:
>> > >>
>> > >> Hello everyone,
>> > >>
>> > >> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a 
>> > >> community.
>> > >>
>> > >> Java 8 is already end of life for commercial users, and many companies 
>> > >> are moving to Java 11.
>> > >> The release date for Apache Spark 3.0 is still not there yet, and there 
>> > >> are many API
>> > >> incompatibility issues when upgrading from Spark 2.x. As a result, 
>> > >> asking users to move to
>> > >> Spark 3.0 to use JDK 11 is not realistic.
>> > >>
>> > >> Should we backport PRs for JDK11 and cut a release in 2.x to support 
>> > >> JDK11?
>> > >>
>> > >> Should we cut a new Apache Spark 2.5 since the patches involve some of 
>> > >> the dependencies changes
>> > >> which is not desired in minor release?
>> > >>
>> > >> Thanks.
>> > >>
>> > >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   
>> > >> Apple, Inc
>> > >>
>> > >>
>> > >> -
>> > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> > >>
>> > >
>> > > -
>> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> > >
>> >
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>>
>> -