Re: StackOverflow in Spark

2016-06-13 Thread Terry Hoo
ag 2 juni 2016 06:06 > *Aan:* Yash Sharma <yash...@gmail.com> > *CC:* Ted Yu <yuzhih...@gmail.com>; Matthew Young <taige...@gmail.com>; > Michel Hubert <mich...@phact.nl>; user@spark.apache.org > *Onderwerp:* Re: StackOverflow in Spark > > > > Stackoverflow

RE: StackOverflow in Spark

2016-06-13 Thread Michel Hubert
gt;; Matthew Young <taige...@gmail.com>; Michel Hubert <mich...@phact.nl>; user@spark.apache.org Onderwerp: Re: StackOverflow in Spark Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and

Re: StackOverflow in Spark

2016-06-01 Thread Rishabh Wadhawan
Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope it helps. Let me know if you need anymore

Re: StackOverflow in Spark

2016-06-01 Thread Yash Sharma
Not sure if its related, But I got a similar stack overflow error some time back while reading files and converting them to parquet. > Stack trace- > 16/06/02 02:23:54 INFO YarnAllocator: Driver requested a total number of > 32769 executor(s). > 16/06/02 02:23:54 INFO ExecutorAllocationManager:

Re: StackOverflow in Spark

2016-06-01 Thread Matthew Young
Hi, It's related to the one fixed bug in Spark, jira ticket SPARK-6847 Matthew Yang On Wed, May 25, 2016 at 7:48 PM, Michel Hubert wrote: > > > Hi, > > > > > > I have an Spark application which generates StackOverflowError