Straw poll: dropping support for things like Scala 2.10

I'd like to gauge where people stand on the issue of dropping support for a few things that were considered for 2.0. First: Scala 2.10. We've seen a number of build breakages this week because the PR builder only tests 2.11. No big deal at this stage, but, it did cause me to wonder whether it's ti

Re: Straw poll: dropping support for things like Scala 2.10

What's changed since the last time we discussed these issues, about 7 months ago? Or, another way to formulate the question: What are the threshold criteria that we should use to decide when to end Scala 2.10 and/or Java 7 support? On Tue, Oct 25, 2016 at 8:36 AM, Sean Owen wrote: > I'd like to

Re: Straw poll: dropping support for things like Scala 2.10

I'd also like to add Python 2.6 to the list of things. We've considered dropping it before but never followed through to the best of my knowledge (although on mobile right now so can't double check). On Tuesday, October 25, 2016, Sean Owen wrote: > I'd like to gauge where people stand on the iss

Re: Straw poll: dropping support for things like Scala 2.10

The general forces are that new versions of things to support emerge, and are valuable to support, but have some cost to support in addition to old versions. And the old versions become less used and therefore less valuable to support, and at some point it tips to being more cost than value. It's h

Re: Straw poll: dropping support for things like Scala 2.10

I think that 2.1 should include a visible deprecation message about Java 7, Scala 2.10 and older Hadoop versions (plus python if there is a consensus on that), to give users / admins early warning, followed by dropping them from trunk for 2.2 once 2.1 is released. Personally, we use only Scala 2.11

Re: Straw poll: dropping support for things like Scala 2.10

I think only supporting 1 version of scala at any given time is not sufficient, 2 probably is ok. I.e. don't drop 2.10 before 2.12 is out + supported On Tue, Oct 25, 2016 at 10:56 AM, Sean Owen wrote: > The general forces are that new versions of things to support emerge, and > are valuable to s

Re: Straw poll: dropping support for things like Scala 2.10

After support is dropped for Java 7, can we have encoders for java.time classes (e.g. LocalDate)? If so, then please drop support for Java 7 ASAP. :-)

Re: Straw poll: dropping support for things like Scala 2.10

it will take time before all libraries that spark depends on are available for scala 2.12, so we are not talking spark 2.1.x and probably also not 2.2.x for scala 2.12 it technically makes sense to drop java 7 and scala 2.10 around the same time as scala 2.12 is introduced we are still heavily de

Re: Straw poll: dropping support for things like Scala 2.10

FYI: Support for both Python 2.6 and Java 7 was deprecated in 2.0 (see release notes under Deprecations). The deprecation notice didn't offer a specific timeline for completely dropping support other than to say they "might be removed in f

Re: Straw poll: dropping support for things like Scala 2.10

No, I think our intent is that using a deprecated language version can generate warnings, but that it should still work; whereas once we remove support for a language version, then it really is ok for Spark developers to do things not compatible with that version and for users attempting to use tha

Re: Straw poll: dropping support for things like Scala 2.10

No, I think our intent is that using a deprecated language version can generate warnings, but that it should still work; whereas once we remove support for a language version, then it really is ok for Spark developers to do things not compatible with that version and for users attempting to use tha

Re: Straw poll: dropping support for things like Scala 2.10

You're right; so we could remove Java 7 support in 2.1.0. Both Holden and I not having the facts immediately to mind does suggest, however, that we should be doing a better job of making sure that information about deprecated language versions is inescapably public. That's harder to do with a lang

Re: Straw poll: dropping support for things like Scala 2.10

Agreed. Would an announcement/reminder on the dev and user lists suffice in this case? Basically, just point out what's already been mentioned in the 2.0 release notes, and include a link there so people know what we're referencing. 2016년 10월 25일 (화) 오후 5:32, Mark Hamstra 님이 작성: > You're right; so

Re: Straw poll: dropping support for things like Scala 2.10

Hi, All. It's great since it's a progress. Then, at least, in 2017, Spark 2.2.0 will be out with JDK8 and Scala 2.11/2.12, right? Bests, Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Straw poll: dropping support for things like Scala 2.10

Is the deprecation of JDK 7 and Scala 2.10 documented anywhere outside the release notes for Spark 2.0.0? I do not consider release notes to be sufficient public notice for deprecation of supported platforms - this should be noted in the documentation somewhere. Here are on the only mentions I coul

Re: Straw poll: dropping support for things like Scala 2.10

Hi, Daniel. I guess that kind of works will start sufficiently in 2.1.0 after PMC's annoucement/reminder on mailing list. Bests, Dongjoon. On Wednesday, October 26, 2016, Daniel Siegmann < dsiegm...@securityscorecard.io> wrote: > Is the deprecation of JDK 7 and Scala 2.10 documented anywhere o

Re: Straw poll: dropping support for things like Scala 2.10

We can do the following concrete proposal: 1. Plan to remove support for Java 7 / Scala 2.10 in Spark 2.2.0 (Mar/Apr 2017). 2. In Spark 2.1.0 release, aggressively and explicitly announce the deprecation of Java 7 / Scala 2.10 support. (a) It should appear in release notes, documentations that m

Re: Straw poll: dropping support for things like Scala 2.10

that sounds good to me On Wed, Oct 26, 2016 at 2:26 PM, Reynold Xin wrote: > We can do the following concrete proposal: > > 1. Plan to remove support for Java 7 / Scala 2.10 in Spark 2.2.0 (Mar/Apr > 2017). > > 2. In Spark 2.1.0 release, aggressively and explicitly announce the > deprecation of

Re: Straw poll: dropping support for things like Scala 2.10

+1 On Wed, Oct 26, 2016 at 11:26 AM, Reynold Xin wrote: > We can do the following concrete proposal: > > 1. Plan to remove support for Java 7 / Scala 2.10 in Spark 2.2.0 (Mar/Apr > 2017). > > 2. In Spark 2.1.0 release, aggressively and explicitly announce the > deprecation of Java 7 / Scala 2.10

Re: Straw poll: dropping support for things like Scala 2.10

Seems OK by me. How about Hadoop < 2.6, Python 2.6? Those seem more removeable. I'd like to add that to a list of things that will begin to be unsupported 6 months from now. On Wed, Oct 26, 2016 at 8:49 PM Koert Kuipers wrote: > that sounds good to me > > On Wed, Oct 26, 2016 at 2:26 PM, Reynold

Re: Straw poll: dropping support for things like Scala 2.10

On 27 Oct 2016, at 10:03, Sean Owen mailto:so...@cloudera.com>> wrote: Seems OK by me. How about Hadoop < 2.6, Python 2.6? Those seem more removeable. I'd like to add that to a list of things that will begin to be unsupported 6 months from now. If you go to java 8 only, then hadoop 2.6+ is ma

Re: Straw poll: dropping support for things like Scala 2.10

I created a JIRA ticket to track this: https://issues.apache.org/jira/browse/SPARK-18138 On Thu, Oct 27, 2016 at 10:19 AM, Steve Loughran wrote: > > On 27 Oct 2016, at 10:03, Sean Owen wrote: > > Seems OK by me. > How about Hadoop < 2.6, Python 2.6? Those seem more removeable. I'd like > to a

Re: Straw poll: dropping support for things like Scala 2.10

+1 On Thu, Oct 27, 2016 at 3:15 AM, Reynold Xin wrote: > I created a JIRA ticket to track this: https://issues.apache. > org/jira/browse/SPARK-18138 > > > > On Thu, Oct 27, 2016 at 10:19 AM, Steve Loughran > wrote: > >> >> On 27 Oct 2016, at 10:03, Sean Owen wrote: >> >> Seems OK by me. >> How

Re: Straw poll: dropping support for things like Scala 2.10

Just to comment on this, I'm generally against removing these types of things unless they create a substantial burden on project contributors. It doesn't sound like Python 2.6 and Java 7 do that yet -- Scala 2.10 might, but then of course we need to wait for 2.12 to be out and stable. In genera

Re: Straw poll: dropping support for things like Scala 2.10

+1 for Matei's point. On Thursday, October 27, 2016, Matei Zaharia wrote: > Just to comment on this, I'm generally against removing these types of > things unless they create a substantial burden on project contributors. It > doesn't sound like Python 2.6 and Java 7 do that yet -- Scala 2.10 mig

Re: Straw poll: dropping support for things like Scala 2.10

+1 for Matei's point. On Thu, Oct 27, 2016 at 8:36 AM, Matei Zaharia wrote: > Just to comment on this, I'm generally against removing these types of > things unless they create a substantial burden on project contributors. It > doesn't sound like Python 2.6 and Java 7 do that yet -- Scala 2.10 mi

Re: Straw poll: dropping support for things like Scala 2.10

+1 on Matei's. _ From: Davies Liu mailto:dav...@databricks.com>> Sent: Thursday, October 27, 2016 9:58 AM Subject: Re: Straw poll: dropping support for things like Scala 2.10 To: Matei Zaharia mailto:matei.zaha...@gmail.com>> Cc: Reynold Xin mailto:r.

Re: Straw poll: dropping support for things like Scala 2.10

The burden may be a little more apparent when dealing with the day to day merging and fixing of breaks. The upside is maybe the more compelling argument though. For example, lambda-fying all the Java code, supporting java.time, and taking advantage of some newer Hadoop/YARN APIs is a moderate win f

Re: Straw poll: dropping support for things like Scala 2.10

I totally agree with Sean, just a small correction: Java 7 and Python 2.6 are already deprecated since Spark 2.0 (after a lengthy discussion), so there is no need to discuss whether they should become deprecated in 2.1 http://spark.apache.org/releases/spark-release-2-0-0.html#deprecations The dis

Re: Straw poll: dropping support for things like Scala 2.10

Deprecating them is fine (and I know they're already deprecated), the question is just whether to remove them. For example, what exactly is the downside of having Python 2.6 or Java 7 right now? If it's high, then we can remove them, but I just haven't seen a ton of details. It also sounded like

Re: Straw poll: dropping support for things like Scala 2.10

BTW maybe one key point that isn't obvious is that with YARN and Mesos, the version of Spark used can be solely up to the developer who writes an app, not to the cluster administrator. So even in very conservative orgs, developers can download a new version of Spark, run it, and demonstrate valu

Re: Straw poll: dropping support for things like Scala 2.10

If the subtext is vendors, then I'd have a look at what recent distros look like. I'll write about CDH as a representative example, but I think other distros are naturally similar. CDH has been on Java 8, Hadoop 2.6, Python 2.7 for almost two years (CDH 5.3 / Dec 2014). Granted, this depends on in

Re: Straw poll: dropping support for things like Scala 2.10

i seem to remember a large spark user (tencent, i believe) chiming in late during these discussions 6-12 months ago and squashing any sort of deprecation given the massive effort that would be required to upgrade their environment. i just want to make sure these convos take into consideration la

Re: Straw poll: dropping support for things like Scala 2.10

Twitter just led the release of Hadoop 2.6.5 precisely because they wanted to keep a Java 6 cluster up: the bigger your cluster, the less of a rush to upgrade. HDP? I believe we install & prefer (openjdk) Java 8, but the Hadoop branch-2 line is intended to build/run on Java 7 too. There's alway

Re: Straw poll: dropping support for things like Scala 2.10

thats correct in my experience: we have found a scala update to be straightforward and basically somewhat invisible to ops, but a java upgrade a pain because it is managed and "certified" by ops. On Fri, Oct 28, 2016 at 9:44 AM, Steve Loughran wrote: > Twitter just led the release of Hadoop 2.6.

Re: Straw poll: dropping support for things like Scala 2.10

I want to bring up the issue of Scala 2.10 support again, to see how people feel about it. Key opinions from the previous responses, I think: Cody: only drop 2.10 support when 2.12 support is added Koert: we need all dependencies to support 2.12; Scala updates are pretty transparent to IT/ops Ofir

Re: Straw poll: dropping support for things like Scala 2.10

given the issues with scala 2.10 and java 8 i am in favor of dropping scala 2.10 in next release On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen wrote: > I want to bring up the issue of Scala 2.10 support again, to see how > people feel about it. Key opinions from the previous responses, I think: > >

Re: Straw poll: dropping support for things like Scala 2.10

+1 on removing 2.10 On Thu, Mar 2, 2017 at 8:51 AM Koert Kuipers wrote: given the issues with scala 2.10 and java 8 i am in favor of dropping scala 2.10 in next release On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen wrote: I want to bring up the issue of Scala 2.10 support again, to see how peop

Re: Straw poll: dropping support for things like Scala 2.10

Let's track further discussion at https://issues.apache.org/jira/browse/SPARK-19810 I am also in favor of removing Scala 2.10 support, and will open a WIP to discuss the change, but am not yet sure whether there are objections or deeper support for this. On Thu, Mar 2, 2017 at 7:51 PM Russell Spi