Re: OOM in spark driver

Or you can increase the driver heap space (export _JAVA_OPTIONS="-Xmx5g") Thanks Best Regards On Wed, Sep 2, 2015 at 11:57 PM, Mike Hynes <91m...@gmail.com> wrote: > Just a thought; this has worked for me before on standalone client > with a similar OOM error in a driver thread. Try setting: >

Re: [VOTE] Release Apache Spark 1.5.0 (RC3)

Thanks Tom. Interestingly it happened between RC2 and RC3. Now my vote is +1/2 unless the memory error is known and has a workaround. Cheers On Fri, Sep 4, 2015 at 7:30 AM, Tom Graves wrote: > The upper/lower case thing is known. >

Re: [VOTE] Release Apache Spark 1.5.0 (RC3)

Hi Krishna, Can you share your code to reproduce the memory allocation issue? Thanks, Yin On Fri, Sep 4, 2015 at 8:00 AM, Krishna Sankar wrote: > Thanks Tom. Interestingly it happened between RC2 and RC3. > Now my vote is +1/2 unless the memory error is known and has a

Re: [VOTE] Release Apache Spark 1.5.0 (RC3)

Krishna - I think the rename happened before rc1 actually. Was done couple months ago. On Fri, Sep 4, 2015 at 5:00 AM, Krishna Sankar wrote: > Thanks Tom. Interestingly it happened between RC2 and RC3. > Now my vote is +1/2 unless the memory error is known and has a

(Spark SQL) partition-scoped UDF

Transformers in Spark ML typically operate on a per-row basis, based on callUDF. For a new transformer that I'm developing, I have a need to transform an entire partition with a function, as opposed to transforming each row separately. The reason is that, in my case, rows must be transformed

Re: Flaky test in DAGSchedulerSuite?

raised https://issues.apache.org/jira/browse/SPARK-10454 and PR On 4 September 2015 at 21:24, Pete Robbins wrote: > I've also just hit this and was about to raise a JIRA for this if there > isn't one already. I have a simple fix. > > On 4 September 2015 at 19:09, Cheolsoo

Re: [VOTE] Release Apache Spark 1.5.0 (RC3)

Yin, It is the https://github.com/xsankar/global-bd-conf/blob/master/004-Orders.ipynb. Cheers On Fri, Sep 4, 2015 at 9:58 AM, Yin Huai wrote: > Hi Krishna, > > Can you share your code to reproduce the memory allocation issue? > > Thanks, > > Yin > > On Fri, Sep 4, 2015

Re: Flaky test in DAGSchedulerSuite?

I've also just hit this and was about to raise a JIRA for this if there isn't one already. I have a simple fix. On 4 September 2015 at 19:09, Cheolsoo Park wrote: > Hi devs, > > I noticed this test case fails intermittently in Jenkins. > > For eg, see the following builds-

Flaky test in DAGSchedulerSuite?

Hi devs, I noticed this test case fails intermittently in Jenkins. For eg, see the following builds- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41991/ https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41999/ The test failed in different PRs, and the

[build system] java package updates on the amplab jenkins workers

i've installed the latest java 7 and 8 packages on all of the jenkins workers! i haven't updated the /usr/java/latest and /usr/java/default symlinks to point to the new java 7 package, as i'd like to wait for downtime when no builds are running. switching java versions mid-build might be fun,

Re: Flaky test in DAGSchedulerSuite?

Thank you Pete! On Fri, Sep 4, 2015 at 1:40 PM, Pete Robbins wrote: > raised https://issues.apache.org/jira/browse/SPARK-10454 and PR > > On 4 September 2015 at 21:24, Pete Robbins wrote: > >> I've also just hit this and was about to raise a JIRA for

Re: Flaky test in DAGSchedulerSuite?

(merge into master, thanks for the quick fix Pete). 2015-09-04 15:58 GMT-07:00 Cheolsoo Park : > Thank you Pete! > > On Fri, Sep 4, 2015 at 1:40 PM, Pete Robbins wrote: > >> raised https://issues.apache.org/jira/browse/SPARK-10454 and PR >> >> On 4