The long lineage causes a long/deep Java object tree (DAG of RDD objects), which needs to be serialized as part of the task creation. When serializing, the whole object DAG needs to be traversed leading to the stackoverflow error.
TD On Mon, Aug 11, 2014 at 7:14 PM, randylu <randyl...@gmail.com> wrote: > hi, TD. I also fall into the trap of long lineage, and your suggestions do > work well. But i don't understand why the long lineage can cause stackover, > and where it takes effect? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-StackOverflowError-when-calling-count-tp5649p11941.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
