[ https://issues.apache.org/jira/browse/SYSTEMML-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias Boehm resolved SYSTEMML-1543. -------------------------------------- Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 0.14 > Parfor remote result merge fails w/ stackoverflowerror from rdd union > --------------------------------------------------------------------- > > Key: SYSTEMML-1543 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1543 > Project: SystemML > Issue Type: Bug > Reporter: Matthias Boehm > Assignee: Matthias Boehm > Priority: Blocker > Fix For: SystemML 0.14 > > > The current parfor remote_spark result merge is implemented as (1) a union > over all output files from individual parfor tasks, and (2) a final > aggregation with or without compare. For many parfor tasks, the union of such > many RDDs leads to stack overflow errors as shown below: > {code} > Exception in thread "main" java.lang.StackOverflowError > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at scala.collection.immutable.List.foreach(List.scala:381) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.immutable.List.map(List.scala:285) > at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:84) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250) > at scala.Option.getOrElse(Option.scala:121) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at scala.collection.immutable.List.foreach(List.scala:381) > {code} > This did not show up before because the parfor optimizer usually compiles > in-memory result merge operations if the outputs are moderately small, i.e., > they fit into the driver. -- This message was sent by Atlassian JIRA (v6.3.15#6346)