[MLlib] StackOverflow Error spark mllib 1.6.1 FpGrowth Algorithm for Association rule generation

Hi, I am getting stackoverflow error when I run FpGrowth algorithm on my 21 million transactions with a low support, since I want almost every products association with other product. I know the problem is caused by the recursive lineage of the algorithm, but I don't know how to get around this

StackOverflow Error spark mllib 1.6.1 FpGrowth Algorithm for Association rule generation

Hi, I am getting stackoverflow error when I run FpGrowth algorithm on my 21 million transactions with a low support, since I want almost every products association with other product. I know the problem is caused by the recursive lineage of the algorithm, but I don't know how to get around this

Re: Looking for help about stackoverflow in spark

about 90+ embedded documents in > it.) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Looking-for-help-about-stackoverflow-in-spark-tp27255.html > Sent from the Apache Spark User List mailing list archiv

Looking for help about stackoverflow in spark

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Looking-for-help-about-stackoverflow-in-spark-tp27255.html Sent from the Apache Spark User List mailing list archive at Nabble.com. -

Re: StackOverflow in Spark

ag 2 juni 2016 06:06 > *Aan:* Yash Sharma <yash...@gmail.com> > *CC:* Ted Yu <yuzhih...@gmail.com>; Matthew Young <taige...@gmail.com>; > Michel Hubert <mich...@phact.nl>; user@spark.apache.org > *Onderwerp:* Re: StackOverflow in Spark > > > > Stackoverflow

RE: StackOverflow in Spark

gt;; Matthew Young <taige...@gmail.com>; Michel Hubert <mich...@phact.nl>; user@spark.apache.org Onderwerp: Re: StackOverflow in Spark Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and

Re: StackOverflow in Spark

Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope it helps. Let me know if you need anymore

Re: StackOverflow in Spark

Not sure if its related, But I got a similar stack overflow error some time back while reading files and converting them to parquet. > Stack trace- > 16/06/02 02:23:54 INFO YarnAllocator: Driver requested a total number of > 32769 executor(s). > 16/06/02 02:23:54 INFO ExecutorAllocationManager:

Re: StackOverflow in Spark

Hi, It's related to the one fixed bug in Spark, jira ticket SPARK-6847 Matthew Yang On Wed, May 25, 2016 at 7:48 PM, Michel Hubert wrote: > > > Hi, > > > > > > I have an Spark application which generates StackOverflowError

StackOverflow in Spark

Hi, I have an Spark application which generates StackOverflowError exceptions after 30+ min. Anyone any ideas? Seems like problems with deserialization of checkpoint data? 16/05/25 10:48:51 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 55449.0 (TID 5584,