Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread Felix Cheung
3.4 is end of life but 3.5 is not. From your link we expect to release Python 3.5.8 around September 2019. From: shane knapp Sent: Thursday, March 28, 2019 7:54 PM To: Hyukjin Kwon Cc: Bryan Cutler; dev; Felix Cheung Subject: Re: Upgrading minimal PyArrow

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread shane knapp
looks like the same for 3.5... https://www.python.org/dev/peps/pep-0478/ let's pick a python version and start testing. On Thu, Mar 28, 2019 at 7:52 PM shane knapp wrote: > >> If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to >> 3.5. >> >> this is inevitable. 3.4s

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread shane knapp
> > > If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to > 3.5. > > this is inevitable. 3.4s final release was 10 days ago ( https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread Hyukjin Kwon
Bryan, was there an actual change when to drop Python 3.4 in PyArrow? If not, I think it might be possible that we can increase the minimal Arrow version separately. If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to 3.5. 2019년 3월 29일 (금) 오전 1:39, Felix Cheung 님이 작성: >

Re: [VOTE] Release Apache Spark 2.4.1 (RC9)

2019-03-28 Thread Marcelo Vanzin
(Anybody knows what's the deal with all the .invalid e-mail addresses?) Anyway. ASF has voting rules, and some things like releases follow specific rules: https://www.apache.org/foundation/voting.html#ReleaseVotes So, for releases, ultimately, the only votes that "count" towards the final tally

Re: [VOTE] Release Apache Spark 2.4.1 (RC9)

2019-03-28 Thread Jonatan Jäderberg
+1 (user vote) btw what to call a vote that is not pmc or committer? Some people use "non-binding”, but nobody says “my vote is binding”, and if some vote is important to me, I still need to look up the who’s-who of the project to be able to tally the votes. I like `user vote` for someone who

[DISCUSS] Enable blacklisting feature by default in 3.0

2019-03-28 Thread Ankur Gupta
Hi all, This is a follow-on to my PR: https://github.com/apache/spark/pull/24208, where I aimed to enable blacklisting for fetch failure by default. From the comments, there is interest in the community to enable overall blacklisting feature by default. I have listed down 3 different things that

Re: [VOTE] Release Apache Spark 2.4.1 (RC9)

2019-03-28 Thread DB Tsai
+1 from myself On Thu, Mar 28, 2019 at 3:14 AM Mihaly Toth wrote: > +1 (non-binding) > > Thanks, Misi > > Sean Owen ezt írta (időpont: 2019. márc. 28., Cs, > 0:19): > >> +1 from me - same as last time. >> >> On Wed, Mar 27, 2019 at 1:31 PM DB Tsai wrote: >> > >> > Please vote on releasing the

[k8s][jenkins] spark dev tool docs now have k8s+minikube instructions!

2019-03-28 Thread shane knapp
https://spark.apache.org/developer-tools.html search for "Testing K8S". this is pretty much how i build and test PRs locally... the commands there are lifted straight from the k8s integration test jenkins build, so they might require a little tweaking to better suit your laptop/server. k8s is

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread Felix Cheung
That’s not necessarily bad. I don’t know if we have plan to ever release any new 2.2.x, 2.3.x at this point and we can message this “supported version” of python change for any new 2.4 release. Besides we could still support python 3.4 - it’s just more complicated to test manually without

Uncaught Exception Handler in master

2019-03-28 Thread Alessandro Liparoti
Hi everyone, I have a spark libary where I would like to do some action before an uncaught exception happens (log it, increment an error metric, ...). I tried multiple times to use setUncaughtExceptionHandler in the current Thread but this doesn't work. If I spawn another thread this works fine.

Re: [Spark SQL]: looking for place operators apply on the dataset / dataframe

2019-03-28 Thread Marco Gaido
Hi, you can check your execution plan and you can find from there which *Exec classes are used. Please notice that in case of wholeStageCodegen, its children operators are executed inside the wholeStageCodegenExec. Bests, Marco Il giorno gio 28 mar 2019 alle ore 15:21 ehsan shams <

Re: [Spark SQL]: looking for place operators apply on the dataset / dataframe

2019-03-28 Thread Sean Owen
I'd suggest loading the source in an IDE if you want to explore the code base. It will let you answer this in one click. Here it's Dataset, as a DataFrame is a Dataset[Row]. On Thu, Mar 28, 2019 at 9:21 AM ehsan shams wrote: > > Hi > > I would like to know where exactly(which class/function)

[Spark SQL]: looking for place operators apply on the dataset / dataframe

2019-03-28 Thread ehsan shams
Hi I would like to know where exactly(which class/function) spark sql will apply the operators on dataset / dataframe rows. For example by applying the following filter or groupby which class is responsible for? And will iterate over the rows to do its operation? Kind regards Ehsan Shams val

Re: [VOTE] Release Apache Spark 2.4.1 (RC9)

2019-03-28 Thread Mihaly Toth
+1 (non-binding) Thanks, Misi Sean Owen ezt írta (időpont: 2019. márc. 28., Cs, 0:19): > +1 from me - same as last time. > > On Wed, Mar 27, 2019 at 1:31 PM DB Tsai wrote: > > > > Please vote on releasing the following candidate as Apache Spark version > 2.4.1. > > > > The vote is open until

Re: Support SqlStreaming in spark

2019-03-28 Thread uncleGen
Hi all, I have rewritten the design doc based on previous discussing. https://docs.google.com/document/d/19degwnIIcuMSELv6BQ_1VQI5AIVcvGeqOm5xE2-aRA0 Would be interested to hear what others think. Regards, Genmao Yu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Support SqlStreaming in spark

2019-03-28 Thread uncleGen
Hi all, I have rewritten the design doc based on previous discussing. https://docs.google.com/document/d/19degwnIIcuMSELv6BQ_1VQI5AIVcvGeqOm5xE2-aRA0 Would be interested to hear what others think. Regards, Genmao Yu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/