The long lineage causes a long/deep Java object tree (DAG of RDD objects), which needs to be serialized as part of the task creation. When serializing, the whole object DAG needs to be traversed leading to the stackoverflow error.
TD On Mon, Aug 11, 2014 at 7:14 PM, randylu <[email protected]> wrote: > hi, TD. I also fall into the trap of long lineage, and your suggestions do > work well. But i don't understand why the long lineage can cause stackover, > and where it takes effect? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-StackOverflowError-when-calling-count-tp5649p11941.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
