Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-02 Thread shane knapp
>> just saying the next release. >>>> >>>> In any case I think in the next release it will be great to get more >>>> Python 3.x release test coverage. >>>> >>>> >>>> >>>> -

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-02 Thread Bryan Cutler
st saying the next release. >>> >>> In any case I think in the next release it will be great to get more >>> Python 3.x release test coverage. >>> >>> >>> >>> -------------- >>> *From:* shane knapp >>>

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-01 Thread shane knapp
I think in the next release it will be great to get more >> Python 3.x release test coverage. >> >> >> >> -- >> *From:* shane knapp >> *Sent:* Friday, March 29, 2019 4:46 PM >> *To:* Bryan Cutler >> *Cc:* Felix Cheun

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-01 Thread shane knapp
3.4 is end of life but 3.5 is not. From your link >>> >>> we expect to release Python 3.5.8 around September 2019. >>> >>> >>> >>> ------ >>> *From:* shane knapp >>> *Sent:* Thursday, March 28, 2019 7:5

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-29 Thread Felix Cheung
To: Bryan Cutler Cc: Felix Cheung; Hyukjin Kwon; dev Subject: Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] i'm not opposed to 3.6 at all. On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler mailto:cutl...@gmail.com>> wrote: PyArrow dropping Python 3.4 was mainly due to support goin

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-29 Thread shane knapp
>> >> >> -- >> *From:* shane knapp >> *Sent:* Thursday, March 28, 2019 7:54 PM >> *To:* Hyukjin Kwon >> *Cc:* Bryan Cutler; dev; Felix Cheung >> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] &g

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-29 Thread Bryan Cutler
t; *Cc:* Bryan Cutler; dev; Felix Cheung > *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] > > looks like the same for 3.5... https://www.python.org/dev/peps/pep-0478/ > > let's pick a python version and start testing. > > On Thu, Mar 28, 2019 at 7:52

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread Felix Cheung
version to 0.12.x [SPARK-27276] looks like the same for 3.5... https://www.python.org/dev/peps/pep-0478/ let's pick a python version and start testing. On Thu, Mar 28, 2019 at 7:52 PM shane knapp mailto:skn...@berkeley.edu>> wrote: If there was, it looks inevitable to upgrade Jenkins\s

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread shane knapp
looks like the same for 3.5... https://www.python.org/dev/peps/pep-0478/ let's pick a python version and start testing. On Thu, Mar 28, 2019 at 7:52 PM shane knapp wrote: > >> If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to >> 3.5. >> >> this is inevitable. 3.4s

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread shane knapp
> > > If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to > 3.5. > > this is inevitable. 3.4s final release was 10 days ago ( https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread Hyukjin Kwon
t manually without Jenkins coverage. > > > -- > *From:* shane knapp > *Sent:* Tuesday, March 26, 2019 12:11 PM > *To:* Bryan Cutler > *Cc:* dev > *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] > > i'm pretty certain th

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread Felix Cheung
Jenkins coverage. From: shane knapp Sent: Tuesday, March 26, 2019 12:11 PM To: Bryan Cutler Cc: dev Subject: Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] i'm pretty certain that i've got a solid python 3.5 conda environment ready to be deployed

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-26 Thread shane knapp
25, 2019 6:48 PM >>> *To:* Hyukjin Kwon >>> *Cc:* dev; Bryan Cutler; Takuya UESHIN; shane knapp >>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x >>> [SPARK-27276] >>> >>> I don't know a lot about Arrow here, but seems reason

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-26 Thread Bryan Cutler
ch 25, 2019 6:48 PM >> *To:* Hyukjin Kwon >> *Cc:* dev; Bryan Cutler; Takuya UESHIN; shane knapp >> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] >> >> I don't know a lot about Arrow here, but seems reasonable. Is this for >> Spark 3.0 or

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-25 Thread Reynold Xin
Hyukjin Kwon > *Cc:* dev; Bryan Cutler; Takuya UESHIN; shane knapp > *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] >   > I don't know a lot about Arrow here, but seems reasonable. Is this for > Spark 3.0 or for 2.x? Certainly, requiring the latest for Spark

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-25 Thread Felix Cheung
I’m +1 if 3.0 From: Sean Owen Sent: Monday, March 25, 2019 6:48 PM To: Hyukjin Kwon Cc: dev; Bryan Cutler; Takuya UESHIN; shane knapp Subject: Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] I don't know a lot about Arrow here, but seems

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-25 Thread Sean Owen
I don't know a lot about Arrow here, but seems reasonable. Is this for Spark 3.0 or for 2.x? Certainly, requiring the latest for Spark 3 seems right. On Mon, Mar 25, 2019 at 8:17 PM Hyukjin Kwon wrote: > > Hi all, > > We really need to upgrade the minimal version soon. It's actually slowing >

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-25 Thread shane knapp
thanks for the heads up... i'll test deploy this tomorrow and see what gotchas turn up. we may need to upgrade from python 3.4 to 3.5 IIRC. On Mon, Mar 25, 2019 at 6:16 PM Hyukjin Kwon wrote: > Hi all, > > We really need to upgrade the minimal version soon. It's actually slowing > down the

Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-25 Thread Hyukjin Kwon
Hi all, We really need to upgrade the minimal version soon. It's actually slowing down the PySpark dev, for instance, by the overhead that sometimes we need currently to test all multiple matrix of Arrow and Pandas. Also, it currently requires to add some weird hacks or ugly codes. Some bugs