Re: [discuss] ending support for Java 7 in Spark 2.0

Hi Sean, See http://www.oracle.com/technetwork/java/eol-135779.html Java 7 hasn't EOLed yet. If you look at support you can get from Oracle, it's actually goes to 2019. And you can even get more support after that. Spark has always maintained great backward compatibility with other systems, way

Re: [discuss] ending support for Java 7 in Spark 2.0

Following https://github.com/apache/spark/pull/12165#issuecomment-205791222 I'd like to make a point about process and then answer points below. We have this funny system where anyone can propose a change, and any of a few people can veto a change unilaterally. The latter rarely comes up. 9

Re: [discuss] ending support for Java 7 in Spark 2.0

Curveball: Is there a need to use lambdas quite yet? On Mon, Apr 4, 2016 at 10:58 PM, Ofir Manor wrote: > I think that a backup plan could be to announce that JDK7 is deprecated in > Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives > admins

Re: [discuss] ending support for Java 7 in Spark 2.0

I think that a backup plan could be to announce that JDK7 is deprecated in Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives admins enough warning to install JDK8 along side their "main" JDK (or fully migrate to it), while allowing the project to merge JDK8-specific

Re: [discuss] ending support for Java 7 in Spark 2.0

Reynold, Considering the performance improvements you mentioned in your original e-mail and also considering that few other big data projects have already or are in progress of abandoning JDK 7, I think it would benefit Spark if we go with JDK 8.0 only. Are there users that will be less

Re: [discuss] ending support for Java 7 in Spark 2.0

Many open source projects are aggressive, such as Oracle JDK and Ubuntu, But they provide stable commercial supporting. In other words, the enterprises doesn't drop JDK7, might aslo do not drop Spark 1.x to adopt Spark 2.x early version. On Sun, Apr 3, 2016 at 10:29 PM -0700, "Reynold

Re: [discuss] ending support for Java 7 in Spark 2.0

Since my original email, I've talked to a lot more users and looked at what various environments support. It is true that a lot of enterprises, and even some technology companies, are still using Java 7. One thing is that up until this date, users still can't install openjdk 8 on Ubuntu by

RE: [discuss] ending support for Java 7 in Spark 2.0

orks.com> Cc: Reynold Xin <r...@databricks.com>; Koert Kuipers <ko...@tresata.com>; Kostas Sakellis <kos...@cloudera.com>; Marcelo Vanzin <van...@cloudera.com>; dev@spark.apache.org Subject: Re: [discuss] ending support for Java 7 in Spark 2.0 Steve, those are good points,

Re: [discuss] ending support for Java 7 in Spark 2.0

Steve, those are good points, I had forgotten Hadoop had those issues.    We run with jdk 8, hadoop is built for jdk7 compatibility, we are running hadoop 2.7 on our clusters and by the time Spark 2.0 is out I would expected a mix of Hadoop 2.7 and 2.8.  We also don't use spnego. I didn't quite

Re: [discuss] ending support for Java 7 in Spark 2.0

Can I note that if Spark 2.0 is going to be Java 8+ only, then that means Hadoop 2.6.x should be the minimum Hadoop version. https://issues.apache.org/jira/browse/HADOOP-11090 Where things get complicated, is that situation of: Hadoop services on Java 7, Spark on Java 8 in its own JVM I'm

Re: [discuss] ending support for Java 7 in Spark 2.0

+1. Tom On Tuesday, March 29, 2016 1:17 PM, Reynold Xin wrote: They work. On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers wrote: if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd party scala libraries compiled with a

Re: [discuss] ending support for Java 7 in Spark 2.0

They work. On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers wrote: > if scala prior to sbt 2.10.4 didn't support java 8, does that mean that > 3rd party scala libraries compiled with a scala version < 2.10.4 might not > work on java 8? > > > On Mon, Mar 28, 2016 at 7:06 PM,

Re: [discuss] ending support for Java 7 in Spark 2.0

if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd party scala libraries compiled with a scala version < 2.10.4 might not work on java 8? On Mon, Mar 28, 2016 at 7:06 PM, Kostas Sakellis wrote: > Also, +1 on dropping jdk7 in Spark 2.0. > > Kostas >

Re: [discuss] ending support for Java 7 in Spark 2.0

Also, +1 on dropping jdk7 in Spark 2.0. Kostas On Mon, Mar 28, 2016 at 2:01 PM, Marcelo Vanzin wrote: > Finally got some internal feedback on this, and we're ok with > requiring people to deploy jdk8 for 2.0, so +1 too. > > On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende

Re: [discuss] ending support for Java 7 in Spark 2.0

Finally got some internal feedback on this, and we're ok with requiring people to deploy jdk8 for 2.0, so +1 too. On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende wrote: > +1, I also checked with few projects inside IBM that consume Spark and they > seem to be ok with the

Re: [discuss] ending support for Java 7 in Spark 2.0

+1, I also checked with few projects inside IBM that consume Spark and they seem to be ok with the direction of droping JDK 7. On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt wrote: > +1 from Mesosphere > > On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 from Mesosphere On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran wrote: > > > On 25 Mar 2016, at 01:59, Mridul Muralidharan wrote: > > > > Removing compatibility (with jdk, etc) can be done with a major release- > given that 7 has been EOLed a while

Re: [discuss] ending support for Java 7 in Spark 2.0

> On 25 Mar 2016, at 01:59, Mridul Muralidharan wrote: > > Removing compatibility (with jdk, etc) can be done with a major release- > given that 7 has been EOLed a while back and is now unsupported, we have to > decide if we drop support for it in 2.0 or 3.0 (2+ years from

Re: [discuss] ending support for Java 7 in Spark 2.0

i asked around a little, and the general trend at our clients seems to be that they plan to upgrade the clusters to java 8 within the year. so with that in mind i wish this was a little later (i would have preferred a java-8-only spark at the end of year). but since a major spark version only

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 on removing Java 7 and Scala 2.10 support. It looks to be entirely possible to support Java 8 containers in a YARN cluster otherwise running Java 7 (example code for alt JAVA_HOME https://issues.apache.org/jira/secure/attachment/12671739/YARN-1964.patch) so really there should be no big

Re: [discuss] ending support for Java 7 in Spark 2.0

I do agree w.r.t scala 2.10 as well; similar arguments apply (though there is a nuanced diff - source compatibility for scala vs binary compatibility wrt Java) Was there a proposal which did not go through ? Not sure if I missed it. Regards Mridul On Thursday, March 24, 2016, Koert Kuipers

Re: [discuss] ending support for Java 7 in Spark 2.0

I do agree w.r.t scala 2.10 as well; similar arguments apply (though there is a nuanced diff - source compatibility for scala vs binary compatibility wrt Java) Was there a proposal which did not go through ? Not sure if I missed it. Regards Mridul On Thursday, March 24, 2016, Koert Kuipers

Re: [discuss] ending support for Java 7 in Spark 2.0

i think that logic is reasonable, but then the same should also apply to scala 2.10, which is also unmaintained/unsupported at this point (basically has been since march 2015 except for one hotfix due to a license incompatibility) who wants to support scala 2.10 three years after they did the

Re: [discuss] ending support for Java 7 in Spark 2.0

Removing compatibility (with jdk, etc) can be done with a major release- given that 7 has been EOLed a while back and is now unsupported, we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from now). Given the functionality & performance benefits of going to jdk8, future

Re: [discuss] ending support for Java 7 in Spark 2.0

i think marcelo also pointed this out before. its very interesting to hear, i was not aware of that until today. it would mean we would only have to convince a group/client with a cluster to install jdk8 on the nodes, without actually transitioning to it, if i understand it correctly. that would

Re: [discuss] ending support for Java 7 in Spark 2.0

Container Java version can be different from yarn Java version : we run jobs with jdk8 on jdk7 cluster without issues. Regards Mridul On Thursday, March 24, 2016, Koert Kuipers wrote: > i guess what i am saying is that in a yarn world the only hard > restrictions left are

Re: [discuss] ending support for Java 7 in Spark 2.0

the good news is, that from an shared infrastructure perspective, most places have zero scala, so the upgrade is actually very easy. i can see how it would be different for say twitter On Thu, Mar 24, 2016 at 7:50 PM, Reynold Xin wrote: > If you want to go down that

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 4:54 PM, Mark Hamstra wrote: > It's a pain in the ass. Especially if some of your transitive > dependencies never upgraded from 2.10 to 2.11. > Yeah, I'm going to have to agree here. It is not as bad as it was in the 2.9 days, but its still

Re: [discuss] ending support for Java 7 in Spark 2.0

There aren't many such libraries, but there are a few. When faced with one of those dependencies that still doesn't go beyond 2.10, you essentially have the choice of taking on the maintenance burden to bring the library up to date, or you do what is potentially a fairly larger refactoring to use

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin wrote: > If you want to go down that route, you should also ask somebody who has had > experience managing a large organization's applications and try to update > Scala version. I understand both sides. But if you look at what

Re: [discuss] ending support for Java 7 in Spark 2.0

It's a pain in the ass. Especially if some of your transitive dependencies never upgraded from 2.10 to 2.11. On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin wrote: > If you want to go down that route, you should also ask somebody who has > had experience managing a large

Re: [discuss] ending support for Java 7 in Spark 2.0

In addition, with Spark 2.0, we are throwing away binary compatibility anyways so user applications will have to be recompiled. The only argument I can see is for libraries that have already been built on Scala 2.10 that are no longer being maintained. How big of an issue do we think that is?

Re: [discuss] ending support for Java 7 in Spark 2.0

If you want to go down that route, you should also ask somebody who has had experience managing a large organization's applications and try to update Scala version. On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin wrote: > On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin wrote: > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not > binary compatible, whereas JVM 7 and 8 are binary compatible except certain

Re: [discuss] ending support for Java 7 in Spark 2.0

Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not binary compatible, whereas JVM 7 and 8 are binary compatible except certain esoteric cases. On Thu, Mar 24, 2016 at 4:44 PM, Kostas Sakellis

Re: [discuss] ending support for Java 7 in Spark 2.0

If an argument here is the ongoing build/maintenance burden I think we should seriously consider dropping scala 2.10 in Spark 2.0. Supporting scala 2.10 is bigger build/infrastructure burden than supporting jdk7 since you actually have to build different artifacts and test them whereas you can

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky wrote: > You can, but since it's going to be a maintainability issue I would > argue it is in fact a problem. Every thing you choose to support generates a maintenance burden. Support 3 versions of Scala would be a huge

Re: [discuss] ending support for Java 7 in Spark 2.0

I mean from the perspective of someone developing Spark, it makes things more complicated. It's just my point of view, people that actually support Spark deployments may have a different opinion ;) On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky wrote: > You can, but since it's

Re: [discuss] ending support for Java 7 in Spark 2.0

You can, but since it's going to be a maintainability issue I would argue it is in fact a problem. On Thu, Mar 24, 2016 at 2:34 PM, Marcelo Vanzin wrote: > Hi Jakob, > > On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky wrote: >> Reynold's 3rd point is

Re: [discuss] ending support for Java 7 in Spark 2.0

Hi Jakob, On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky wrote: > Reynold's 3rd point is particularly strong in my opinion. Supporting > Consider what would happen if Spark 2.0 doesn't require Java 8 and > hence not support Scala 2.12. Will it be stuck on an older version >

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 for Java 8 only I think it will make it easier to make a unified API for Java and Scala, instead of the wrappers of Java over Scala. On Mar 24, 2016 11:46 AM, "Stephen Boesch" wrote: > +1 for java8 only +1 for 2.11+ only .At this point scala libraries > supporting

Re: [discuss] ending support for Java 7 in Spark 2.0

Reynold's 3rd point is particularly strong in my opinion. Supporting Scala 2.12 will require Java 8 anyway, and introducing such a change is probably best done in a major release. Consider what would happen if Spark 2.0 doesn't require Java 8 and hence not support Scala 2.12. Will it be stuck on

Re: [discuss] ending support for Java 7 in Spark 2.0

Spark 2.x has to be the time for Java 8. I'd rather increase JVM major version on a Spark major version than on a Spark minor version, and I'd rather Spark do that upgrade for the 2.x series than the 3.x series (~2yr from now based on the lifetime of Spark 1.x). If we wait until the next

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 for java8 only +1 for 2.11+ only .At this point scala libraries supporting only 2.10 are typically less active and/or poorly maintained. That trend will only continue when considering the lifespan of spark 2.X. 2016-03-24 11:32 GMT-07:00 Steve Loughran : > > On

Re: [discuss] ending support for Java 7 in Spark 2.0

On 24 Mar 2016, at 15:27, Koert Kuipers > wrote: i think the arguments are convincing, but it also makes me wonder if i live in some kind of alternate universe... we deploy on customers clusters, where the OS, python version, java version and hadoop

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 10:13 AM, Reynold Xin wrote: > Yes So is it safe to say the only hard requirements for Java 8 in your list is (4)? (1) and (3) are infrastructure issues. Yes, it sucks to maintain more testing infrastructure and potentially more complicated build

Re: [discuss] ending support for Java 7 in Spark 2.0

Yes On Thursday, March 24, 2016, Marcelo Vanzin wrote: > On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin > wrote: > > I actually talked quite a bit today with an engineer on the scala > compiler > > team tonight and the scala 2.10 + java 8

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 9:54 AM, Koert Kuipers wrote: > i guess what i am saying is that in a yarn world the only hard restrictions > left are the the containers you run in, which means the hadoop version, java > version and python version (if you use python). It is

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin wrote: > I actually talked quite a bit today with an engineer on the scala compiler > team tonight and the scala 2.10 + java 8 combo should be ok. The latest > Scala 2.10 release should have all the important fixes that are needed

Re: [discuss] ending support for Java 7 in Spark 2.0

i guess what i am saying is that in a yarn world the only hard restrictions left are the the containers you run in, which means the hadoop version, java version and python version (if you use python). On Thu, Mar 24, 2016 at 12:39 PM, Koert Kuipers wrote: > The group will

Re: [discuss] ending support for Java 7 in Spark 2.0

Thank you for the context Jean... I appreciate it... On Thu, Mar 24, 2016 at 12:40 PM, Jean-Baptiste Onofré wrote: > Hi Al, > > Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will go > on Spark 2.0, but maintenance release can be performed on 1.x branch. >

Re: [discuss] ending support for Java 7 in Spark 2.0

Hi Al, Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will go on Spark 2.0, but maintenance release can be performed on 1.x branch. Regards JB On 03/24/2016 05:38 PM, Al Pivonka wrote: As an end user (developer) and Cluster Admin. I would have to agree with Koet. To me

Re: [discuss] ending support for Java 7 in Spark 2.0

The group will not upgrade to spark 2.0 themselves, but they are mostly fine with vendors like us deploying our application via yarn with whatever spark version we choose (and bundle, so they do not install it separately, they might not even be aware of what spark version we use). This all works

Re: [discuss] ending support for Java 7 in Spark 2.0

As an end user (developer) and Cluster Admin. I would have to agree with Koet. To me the real question is timing, current version is 1.6.1, the question I have is how many more releases till 2.0 and what is the time frame? If you give people six to twelve months to plan and make sure they know

Re: [discuss] ending support for Java 7 in Spark 2.0

(PS CDH5 runs fine with Java 8, but I understand your more general point.) This is a familiar context indeed, but in that context, would a group not wanting to update to Java 8 want to manually put Spark 2.0 into the mix? That is, if this is a context where the cluster is purposefully some stable

Re: [discuss] ending support for Java 7 in Spark 2.0

i think the arguments are convincing, but it also makes me wonder if i live in some kind of alternate universe... we deploy on customers clusters, where the OS, python version, java version and hadoop distro are not chosen by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 to support Java 8 (and future) *only* in Spark 2.0, and end support of Java 7. It makes sense. Regards JB On 03/24/2016 08:27 AM, Reynold Xin wrote: About a year ago we decided to drop Java 6 support in Spark 1.5. I am wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.

Re: [discuss] ending support for Java 7 in Spark 2.0

> On 24 Mar 2016, at 07:27, Reynold Xin wrote: > > About a year ago we decided to drop Java 6 support in Spark 1.5. I am > wondering if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark > 2.0 would require Java 8 to run). > > Oracle ended public updates

Re: [discuss] ending support for Java 7 in Spark 2.0

Maybe so; I think we have a ticket open to update to 2.10.6, which maybe fixes it. It brings up a different point: supporting multiple Scala versions is much more painful than Java versions because of mutual incompatibility. Right now I get the sense there's an intent to keep supporting 2.10, and

Re: [discuss] ending support for Java 7 in Spark 2.0

I actually talked quite a bit today with an engineer on the scala compiler team tonight and the scala 2.10 + java 8 combo should be ok. The latest Scala 2.10 release should have all the important fixes that are needed for Java 8. On Thu, Mar 24, 2016 at 1:01 AM, Sean Owen

Re: [discuss] ending support for Java 7 in Spark 2.0

I generally favor this for the simplification. I didn't realize there were actually some performance wins and important bug fixes. I've had lots of trouble with scalac 2.10 + Java 8. I don't know if it's still a problem since 2.11 + 8 seems OK, but for a long time the sql/ modules would never

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 Agree, dropping support for java 7 is long overdue - and 2.0 would be a logical release to do this on. Regards, Mridul On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin wrote: > About a year ago we decided to drop Java 6 support in Spark 1.5. I am > wondering if we should

RE: [discuss] ending support for Java 7 in Spark 2.0

From: Reynold Xin [mailto:r...@databricks.com] Sent: Thursday, March 24, 2016 9:37 AM To: dev@spark.apache.org Subject: Re: [discuss] ending support for Java 7 in Spark 2.0 One other benefit that I didn't mention is that we'd be able to use Java 8's Optional class to replace our built-in Optional.

Re: [discuss] ending support for Java 7 in Spark 2.0

One other benefit that I didn't mention is that we'd be able to use Java 8's Optional class to replace our built-in Optional. On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin wrote: > About a year ago we decided to drop Java 6 support in Spark 1.5. I am > wondering if we

[discuss] ending support for Java 7 in Spark 2.0

About a year ago we decided to drop Java 6 support in Spark 1.5. I am wondering if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark 2.0 would require Java 8 to run). Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and removed public downloads for JDK 7 in July