[ https://issues.apache.org/jira/browse/SPARK-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511689#comment-14511689 ]
Patrick Wendell commented on SPARK-7103: ---------------------------------------- Escalated the priority since IMO this is good to fix. > SparkContext.union crashed when some RDDs have no partitioner > ------------------------------------------------------------- > > Key: SPARK-7103 > URL: https://issues.apache.org/jira/browse/SPARK-7103 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.3.0, 1.3.1 > Reporter: Steven She > Priority: Critical > > I encountered a bug where Spark crashes with the following stack trace: > {noformat} > java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:313) > at scala.None$.get(Option.scala:311) > at > org.apache.spark.rdd.PartitionerAwareUnionRDD.getPartitions(PartitionerAwareUnionRDD.scala:69) > {noformat} > Here's a minimal example that reproduces it on the Spark shell: > {noformat} > val x = sc.parallelize(Seq(1->true,2->true,3->false)).partitionBy(new > HashPartitioner(1)) > val y = sc.parallelize(Seq(1->true)) > sc.union(y, x).count() // crashes > sc.union(x, y).count() // This works since the first RDD has a partitioner > {noformat} > We had to resort to instantiating the UnionRDD directly to avoid the > PartitionerAwareUnionRDD. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org