Since my original email, I've talked to a lot more users and looked at what various environments support. It is true that a lot of enterprises, and even some technology companies, are still using Java 7. One thing is that up until this date, users still can't install openjdk 8 on Ubuntu by default. I see that as an indication that it is too early to drop Java 7.
Looking at the timeline, JDK release a major new version roughly every 3 years. We dropped Java 6 support one year ago, so from a timeline point of view we would be very aggressive here if we were to drop Java 7 support in Spark 2.0. Note that not dropping Java 7 support now doesn't mean we have to support Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even though Spark 1.0 started with Java 6. In terms of testing, Josh has actually improved our test infra so now we would run the Java 8 tests: https://github.com/apache/spark/pull/12073 On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin <lwl...@gmail.com> wrote: > Arguments are really convincing; new Dataset API as well as performance > > improvements is exiting, so I'm personally +1 on moving onto Java8. > > > > However, I'm afraid Tencent is one of "the organizations stuck with Java7" > > -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and > > wouldn't upgrade to Java8 until Java9 is out. > > > So: > > (non-binding) +1 on dropping scala 2.10 support > > (non-binding) -1 on dropping Java 7 support > > * as long as we figure out a practical way to run > Spark with > > JDK8 on JDK7 clusters, this -1 would then > definitely be +1 > > > Thanks ! > > On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers <ko...@tresata.com> wrote: > >> i think that logic is reasonable, but then the same should also apply to >> scala 2.10, which is also unmaintained/unsupported at this point (basically >> has been since march 2015 except for one hotfix due to a license >> incompatibility) >> >> who wants to support scala 2.10 three years after they did the last >> maintenance release? >> >> >> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan <mri...@gmail.com> >> wrote: >> >>> Removing compatibility (with jdk, etc) can be done with a major release- >>> given that 7 has been EOLed a while back and is now unsupported, we have to >>> decide if we drop support for it in 2.0 or 3.0 (2+ years from now). >>> >>> Given the functionality & performance benefits of going to jdk8, future >>> enhancements relevant in 2.x timeframe ( scala, dependencies) which >>> requires it, and simplicity wrt code, test & support it looks like a good >>> checkpoint to drop jdk7 support. >>> >>> As already mentioned in the thread, existing yarn clusters are >>> unaffected if they want to continue running jdk7 and yet use >>> spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case >>> distribute jdk8 as archive - suboptimal). >>> I am unsure about mesos (standalone might be easier upgrade I guess ?). >>> >>> >>> Proposal is for 1.6x line to continue to be supported with critical >>> fixes; newer features will require 2.x and so jdk8 >>> >>> Regards >>> Mridul >>> >>> >>> On Thursday, March 24, 2016, Marcelo Vanzin <van...@cloudera.com> wrote: >>> >>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin <r...@databricks.com> >>>> wrote: >>>> > If you want to go down that route, you should also ask somebody who >>>> has had >>>> > experience managing a large organization's applications and try to >>>> update >>>> > Scala version. >>>> >>>> I understand both sides. But if you look at what I've been asking >>>> since the beginning, it's all about the cost and benefits of dropping >>>> support for java 1.7. >>>> >>>> The biggest argument in your original e-mail is about testing. And the >>>> testing cost is much bigger for supporting scala 2.10 than it is for >>>> supporting java 1.7. If you read one of my earlier replies, it should >>>> be even possible to just do everything in a single job - compile for >>>> java 7 and still be able to test things in 1.8, including lambdas, >>>> which seems to be the main thing you were worried about. >>>> >>>> >>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin <van...@cloudera.com> >>>> wrote: >>>> >> >>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin <r...@databricks.com> >>>> wrote: >>>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than >>>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 >>>> are >>>> >> > not >>>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible except >>>> >> > certain >>>> >> > esoteric cases. >>>> >> >>>> >> True, but ask anyone who manages a large cluster how long it would >>>> >> take them to upgrade the jdk across their cluster and validate all >>>> >> their applications and everything... binary compatibility is a tiny >>>> >> drop in that bucket. >>>> >> >>>> >> -- >>>> >> Marcelo >>>> > >>>> > >>>> >>>> >>>> >>>> -- >>>> Marcelo >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>> >>>> >> >
