Re: Should python-2 be supported in Spark 3.0?

-- > *From:* shane knapp > *Sent:* Friday, May 31, 2019 7:38:10 PM > *To:* Denny Lee > *Cc:* Holden Karau; Bryan Cutler; Erik Erlandson; Felix Cheung; Mark > Hamstra; Matei Zaharia; Reynold Xin; Sean Owen; Wenchen Fen; Xiangrui Meng; > dev; user > *Subj

Re: Should python-2 be supported in Spark 3.0?

. From: shane knapp Sent: Friday, May 31, 2019 7:38:10 PM To: Denny Lee Cc: Holden Karau; Bryan Cutler; Erik Erlandson; Felix Cheung; Mark Hamstra; Matei Zaharia; Reynold Xin; Sean Owen; Wenchen Fen; Xiangrui Meng; dev; user Subject: Re: Should python-2 be supported in Spark 3.0? +1000

Re: Should python-2 be supported in Spark 3.0?

state that Python 2 is deprecated in Spark 3.0 >>>>>> >>>>>> I suspect people will then ask when is Spark 3.0 coming out then. >>>>>> Might need to provide some clarity on that. >>>>>> >>>>> >>>

Re: Should python-2 be supported in Spark 3.0?

te that Python 2 is deprecated in Spark 3.0 >>>>> >>>>> I suspect people will then ask when is Spark 3.0 coming out then. >>>>> Might need to provide some clarity on that. >>>>> >>>> >>>> We can say the "next ma

Re: Should python-2 be supported in Spark 3.0?

say the "next major release in 2019" instead of Spark 3.0. Spark >>> 3.0 timeline certainly requires a new thread to discuss. >>> >>> >>>> >>>> >>>> -- >>>> *From:* Reynold Xin >>>&g

Re: Should python-2 be supported in Spark 3.0?

> need to provide some clarity on that. >>> >> >> We can say the "next major release in 2019" instead of Spark 3.0. Spark >> 3.0 timeline certainly requires a new thread to discuss. >> >> >>> >>> >>> ------ >>> *From:* Reynold

Re: Should python-2 be supported in Spark 3.0?

t:* Thursday, May 30, 2019 12:59:14 AM >> *To:* shane knapp >> *Cc:* Erik Erlandson; Mark Hamstra; Matei Zaharia; Sean Owen; Wenchen >> Fen; Xiangrui Meng; dev; user >> *Subject:* Re: Should python-2 be supported in Spark 3.0? >> >> +1 on Xiangrui’s plan. >

Re: Should python-2 be supported in Spark 3.0?

From:* Reynold Xin > *Sent:* Thursday, May 30, 2019 12:59:14 AM > *To:* shane knapp > *Cc:* Erik Erlandson; Mark Hamstra; Matei Zaharia; Sean Owen; Wenchen > Fen; Xiangrui Meng; dev; user > *Subject:* Re: Should python-2 be supported in Spark 3.0? > > +1 on Xiangrui’s plan. &

Re: Should python-2 be supported in Spark 3.0?

ent: Thursday, May 30, 2019 12:59:14 AM To: shane knapp Cc: Erik Erlandson; Mark Hamstra; Matei Zaharia; Sean Owen; Wenchen Fen; Xiangrui Meng; dev; user Subject: Re: Should python-2 be supported in Spark 3.0? +1 on Xiangrui’s plan. On Thu, May 30, 2019 at 7:55 AM shane knapp mailto:skn...@berkeley.

Re: Should python-2 be supported in Spark 3.0?

+1 on Xiangrui’s plan. On Thu, May 30, 2019 at 7:55 AM shane knapp wrote: > I don't have a good sense of the overhead of continuing to support >> Python 2; is it large enough to consider dropping it in Spark 3.0? >> >> from the build/test side, it will actually be pretty easy to continue >

Re: Should python-2 be supported in Spark 3.0?

> > I don't have a good sense of the overhead of continuing to support > Python 2; is it large enough to consider dropping it in Spark 3.0? > > from the build/test side, it will actually be pretty easy to continue support for python2.7 for spark 2.x as the feature sets won't be expanding. that

Re: Should python-2 be supported in Spark 3.0?

Here’s the tweet from the horse’s mouth: https://twitter.com/gvanrossum/status/1133496146700058626?s=21 Cheers Jules — Sent from my iPhone Pardon the dumb thumb typos :) > On May 29, 2019, at 10:12 PM, Sean Owen wrote: > > Deprecated -- certainly and sooner than later. > I don't have a

Re: Should python-2 be supported in Spark 3.0?

Deprecated -- certainly and sooner than later. I don't have a good sense of the overhead of continuing to support Python 2; is it large enough to consider dropping it in Spark 3.0? On Wed, May 29, 2019 at 11:47 PM Xiangrui Meng wrote: > > Hi all, > > I want to revive this old thread since no

Re: Should python-2 be supported in Spark 3.0?

Hi all, I want to revive this old thread since no action was taken so far. If we plan to mark Python 2 as deprecated in Spark 3.0, we should do it as early as possible and let users know ahead. PySpark depends on Python, numpy, pandas, and pyarrow, all of which are sunsetting Python 2 support by

Re: Should python-2 be supported in Spark 3.0?

I think that makes sense. The main benefit of deprecating *prior* to 3.0 would be informational - making the community aware of the upcoming transition earlier. But there are other ways to start informing the community between now and 3.0, besides formal deprecation. I have some residual

Re: Should python-2 be supported in Spark 3.0?

i'd like to second that. if we want to communicate timeline, we can add to the release notes saying py2 will be deprecated in 3.0, and removed in a 3.x release. -- excuse the brevity and lower case due to wrist injury On Mon, Sep 17, 2018 at 4:24 PM Matei Zaharia wrote: > That’s a good point

Re: Should python-2 be supported in Spark 3.0?

That’s a good point — I’d say there’s just a risk of creating a perception issue. First, some users might feel that this means they have to migrate now, which is before Python itself drops support; they might also be surprised that we did this in a minor release (e.g. might we drop Python 2

Re: Should python-2 be supported in Spark 3.0?

FWIW, Pandas is dropping Py2 support at the end of this year. Tensorflow is less clear. They only support py3 on windows, but there is no reference to any policy about py2 on their roadmap or the

Re: Should python-2 be supported in Spark 3.0?

What is the disadvantage to deprecating now in 2.4.0? I mean, it doesn't change the code at all; it's just a notification that we will eventually cease supporting Py2. Wouldn't users prefer to get that notification sooner rather than later? On Mon, Sep 17, 2018 at 12:58 PM Matei Zaharia wrote:

Re: Should python-2 be supported in Spark 3.0?

I’d like to understand the maintenance burden of Python 2 before deprecating it. Since it is not EOL yet, it might make sense to only deprecate it once it’s EOL (which is still over a year from now). Supporting Python 2+3 seems less burdensome than supporting, say, multiple Scala versions in

Re: Should python-2 be supported in Spark 3.0?

If we're going to do that, then we need to do it right now, since 2.4.0 is already in release candidates. On Mon, Sep 17, 2018 at 10:57 AM Erik Erlandson wrote: > I like Mark’s concept for deprecating Py2 starting with 2.4: It may seem > like a ways off but even now there may be some spark

Re: Should python-2 be supported in Spark 3.0?

I like Mark’s concept for deprecating Py2 starting with 2.4: It may seem like a ways off but even now there may be some spark versions supporting Py2 past the point where Py2 is no longer receiving security patches On Sun, Sep 16, 2018 at 12:26 PM Mark Hamstra wrote: > We could also deprecate

Re: Should python-2 be supported in Spark 3.0?

out >> deprecating it first... >> >> >> -- >> *From:* Mark Hamstra >> *Sent:* Sunday, September 16, 2018 12:26 PM >> *To:* Erik Erlandson >> *Cc:* u...@spark.apache.org; dev >> *Subject:* Re: Should python-2 be supported in Spark 3.0? >> >> We could

Re: Should python-2 be supported in Spark 3.0?

I don’t think we should remove any API even in a major release without deprecating it first... From: Mark Hamstra Sent: Sunday, September 16, 2018 12:26 PM To: Erik Erlandson Cc: u...@spark.apache.org; dev Subject: Re: Should python-2 be supported in Spark 3.0

Re: Should python-2 be supported in Spark 3.0?

We could also deprecate Py2 already in the 2.4.0 release. On Sat, Sep 15, 2018 at 11:46 AM Erik Erlandson wrote: > In case this didn't make it onto this thread: > > There is a 3rd option, which is to deprecate Py2 for Spark-3.0, and remove > it entirely on a later 3.x release. > > On Sat, Sep

Re: Should python-2 be supported in Spark 3.0?

In case this didn't make it onto this thread: There is a 3rd option, which is to deprecate Py2 for Spark-3.0, and remove it entirely on a later 3.x release. On Sat, Sep 15, 2018 at 11:09 AM, Erik Erlandson wrote: > On a separate dev@spark thread, I raised a question of whether or not to >

Re: Should python-2 be supported in Spark 3.0?

As Reynold pointed out, we don't have to drop Python 2 support right off the bat. We can just deprecate it with Spark 3.0, which would allow us to actually drop it at a later 3.x release. On Sat, Sep 15, 2018 at 2:09 PM Erik Erlandson wrote: > On a separate dev@spark thread, I raised a question

Should python-2 be supported in Spark 3.0?

On a separate dev@spark thread, I raised a question of whether or not to support python 2 in Apache Spark, going forward into Spark 3.0. Python-2 is going EOL at the end of 2019. The upcoming release of Spark 3.0 is an opportunity to make breaking