Re: Straw poll: dropping support for things like Scala 2.10

Let's track further discussion at https://issues.apache.org/jira/browse/SPARK-19810 I am also in favor of removing Scala 2.10 support, and will open a WIP to discuss the change, but am not yet sure whether there are objections or deeper support for this. On Thu, Mar 2, 2017 at 7:51 PM Russell

Re: Straw poll: dropping support for things like Scala 2.10

+1 on removing 2.10 On Thu, Mar 2, 2017 at 8:51 AM Koert Kuipers wrote: given the issues with scala 2.10 and java 8 i am in favor of dropping scala 2.10 in next release On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen wrote: I want to bring up the issue of

Re: Straw poll: dropping support for things like Scala 2.10

given the issues with scala 2.10 and java 8 i am in favor of dropping scala 2.10 in next release On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen wrote: > I want to bring up the issue of Scala 2.10 support again, to see how > people feel about it. Key opinions from the previous

Re: Straw poll: dropping support for things like Scala 2.10

I want to bring up the issue of Scala 2.10 support again, to see how people feel about it. Key opinions from the previous responses, I think: Cody: only drop 2.10 support when 2.12 support is added Koert: we need all dependencies to support 2.12; Scala updates are pretty transparent to IT/ops

Re: Straw poll: dropping support for things like Scala 2.10

thats correct in my experience: we have found a scala update to be straightforward and basically somewhat invisible to ops, but a java upgrade a pain because it is managed and "certified" by ops. On Fri, Oct 28, 2016 at 9:44 AM, Steve Loughran wrote: > Twitter just led

Re: Straw poll: dropping support for things like Scala 2.10

i seem to remember a large spark user (tencent, i believe) chiming in late during these discussions 6-12 months ago and squashing any sort of deprecation given the massive effort that would be required to upgrade their environment. i just want to make sure these convos take into consideration

Re: Straw poll: dropping support for things like Scala 2.10

If the subtext is vendors, then I'd have a look at what recent distros look like. I'll write about CDH as a representative example, but I think other distros are naturally similar. CDH has been on Java 8, Hadoop 2.6, Python 2.7 for almost two years (CDH 5.3 / Dec 2014). Granted, this depends on

Re: Straw poll: dropping support for things like Scala 2.10

I totally agree with Sean, just a small correction: Java 7 and Python 2.6 are already deprecated since Spark 2.0 (after a lengthy discussion), so there is no need to discuss whether they should become deprecated in 2.1 http://spark.apache.org/releases/spark-release-2-0-0.html#deprecations The

Re: Straw poll: dropping support for things like Scala 2.10

The burden may be a little more apparent when dealing with the day to day merging and fixing of breaks. The upside is maybe the more compelling argument though. For example, lambda-fying all the Java code, supporting java.time, and taking advantage of some newer Hadoop/YARN APIs is a moderate win

Re: Straw poll: dropping support for things like Scala 2.10

+1 for Matei's point. On Thu, Oct 27, 2016 at 8:36 AM, Matei Zaharia wrote: > Just to comment on this, I'm generally against removing these types of > things unless they create a substantial burden on project contributors. It > doesn't sound like Python 2.6 and Java 7 do

Re: Straw poll: dropping support for things like Scala 2.10

+1 for Matei's point. On Thursday, October 27, 2016, Matei Zaharia wrote: > Just to comment on this, I'm generally against removing these types of > things unless they create a substantial burden on project contributors. It > doesn't sound like Python 2.6 and Java 7 do

Re: Straw poll: dropping support for things like Scala 2.10

Just to comment on this, I'm generally against removing these types of things unless they create a substantial burden on project contributors. It doesn't sound like Python 2.6 and Java 7 do that yet -- Scala 2.10 might, but then of course we need to wait for 2.12 to be out and stable. In

Re: Straw poll: dropping support for things like Scala 2.10

+1 On Thu, Oct 27, 2016 at 3:15 AM, Reynold Xin wrote: > I created a JIRA ticket to track this: https://issues.apache. > org/jira/browse/SPARK-18138 > > > > On Thu, Oct 27, 2016 at 10:19 AM, Steve Loughran > wrote: > >> >> On 27 Oct 2016, at 10:03,

Re: Straw poll: dropping support for things like Scala 2.10

I created a JIRA ticket to track this: https://issues.apache.org/jira/browse/SPARK-18138 On Thu, Oct 27, 2016 at 10:19 AM, Steve Loughran wrote: > > On 27 Oct 2016, at 10:03, Sean Owen wrote: > > Seems OK by me. > How about Hadoop < 2.6, Python

Re: Straw poll: dropping support for things like Scala 2.10

On 27 Oct 2016, at 10:03, Sean Owen > wrote: Seems OK by me. How about Hadoop < 2.6, Python 2.6? Those seem more removeable. I'd like to add that to a list of things that will begin to be unsupported 6 months from now. If you go to java 8 only,

Re: Straw poll: dropping support for things like Scala 2.10

Seems OK by me. How about Hadoop < 2.6, Python 2.6? Those seem more removeable. I'd like to add that to a list of things that will begin to be unsupported 6 months from now. On Wed, Oct 26, 2016 at 8:49 PM Koert Kuipers wrote: > that sounds good to me > > On Wed, Oct 26, 2016

Re: Straw poll: dropping support for things like Scala 2.10

+1 On Wed, Oct 26, 2016 at 11:26 AM, Reynold Xin wrote: > We can do the following concrete proposal: > > 1. Plan to remove support for Java 7 / Scala 2.10 in Spark 2.2.0 (Mar/Apr > 2017). > > 2. In Spark 2.1.0 release, aggressively and explicitly announce the > deprecation

Re: Straw poll: dropping support for things like Scala 2.10

that sounds good to me On Wed, Oct 26, 2016 at 2:26 PM, Reynold Xin wrote: > We can do the following concrete proposal: > > 1. Plan to remove support for Java 7 / Scala 2.10 in Spark 2.2.0 (Mar/Apr > 2017). > > 2. In Spark 2.1.0 release, aggressively and explicitly announce

Re: Straw poll: dropping support for things like Scala 2.10

We can do the following concrete proposal: 1. Plan to remove support for Java 7 / Scala 2.10 in Spark 2.2.0 (Mar/Apr 2017). 2. In Spark 2.1.0 release, aggressively and explicitly announce the deprecation of Java 7 / Scala 2.10 support. (a) It should appear in release notes, documentations that

Re: Straw poll: dropping support for things like Scala 2.10

Hi, All. It's great since it's a progress. Then, at least, in 2017, Spark 2.2.0 will be out with JDK8 and Scala 2.11/2.12, right? Bests, Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Straw poll: dropping support for things like Scala 2.10

Agreed. Would an announcement/reminder on the dev and user lists suffice in this case? Basically, just point out what's already been mentioned in the 2.0 release notes, and include a link there so people know what we're referencing. 2016년 10월 25일 (화) 오후 5:32, Mark Hamstra

Re: Straw poll: dropping support for things like Scala 2.10

You're right; so we could remove Java 7 support in 2.1.0. Both Holden and I not having the facts immediately to mind does suggest, however, that we should be doing a better job of making sure that information about deprecated language versions is inescapably public. That's harder to do with a

Re: Straw poll: dropping support for things like Scala 2.10

No, I think our intent is that using a deprecated language version can generate warnings, but that it should still work; whereas once we remove support for a language version, then it really is ok for Spark developers to do things not compatible with that version and for users attempting to use

Re: Straw poll: dropping support for things like Scala 2.10

FYI: Support for both Python 2.6 and Java 7 was deprecated in 2.0 (see release notes under Deprecations). The deprecation notice didn't offer a specific timeline for completely dropping support other than to say they "might be removed in

Re: Straw poll: dropping support for things like Scala 2.10

it will take time before all libraries that spark depends on are available for scala 2.12, so we are not talking spark 2.1.x and probably also not 2.2.x for scala 2.12 it technically makes sense to drop java 7 and scala 2.10 around the same time as scala 2.12 is introduced we are still heavily

Re: Straw poll: dropping support for things like Scala 2.10

After support is dropped for Java 7, can we have encoders for java.time classes (e.g. LocalDate)? If so, then please drop support for Java 7 ASAP. :-)

Re: Straw poll: dropping support for things like Scala 2.10

I think only supporting 1 version of scala at any given time is not sufficient, 2 probably is ok. I.e. don't drop 2.10 before 2.12 is out + supported On Tue, Oct 25, 2016 at 10:56 AM, Sean Owen wrote: > The general forces are that new versions of things to support emerge,

Re: Straw poll: dropping support for things like Scala 2.10

I think that 2.1 should include a visible deprecation message about Java 7, Scala 2.10 and older Hadoop versions (plus python if there is a consensus on that), to give users / admins early warning, followed by dropping them from trunk for 2.2 once 2.1 is released. Personally, we use only Scala

Re: Straw poll: dropping support for things like Scala 2.10

The general forces are that new versions of things to support emerge, and are valuable to support, but have some cost to support in addition to old versions. And the old versions become less used and therefore less valuable to support, and at some point it tips to being more cost than value. It's

Re: Straw poll: dropping support for things like Scala 2.10

I'd also like to add Python 2.6 to the list of things. We've considered dropping it before but never followed through to the best of my knowledge (although on mobile right now so can't double check). On Tuesday, October 25, 2016, Sean Owen wrote: > I'd like to gauge where

Re: Straw poll: dropping support for things like Scala 2.10

What's changed since the last time we discussed these issues, about 7 months ago? Or, another way to formulate the question: What are the threshold criteria that we should use to decide when to end Scala 2.10 and/or Java 7 support? On Tue, Oct 25, 2016 at 8:36 AM, Sean Owen

Straw poll: dropping support for things like Scala 2.10

I'd like to gauge where people stand on the issue of dropping support for a few things that were considered for 2.0. First: Scala 2.10. We've seen a number of build breakages this week because the PR builder only tests 2.11. No big deal at this stage, but, it did cause me to wonder whether it's