[ https://issues.apache.org/jira/browse/SPARK-25283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-25283: ------------------------------------ Assignee: Apache Spark > A deadlock in UnionRDD > ---------------------- > > Key: SPARK-25283 > URL: https://issues.apache.org/jira/browse/SPARK-25283 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.0 > Reporter: Maxim Gekk > Assignee: Apache Spark > Priority: Major > > The PR https://github.com/apache/spark/pull/21913 replaced Scala parallel > collections in UnionRDD by new parmap function. This changes cause a deadlock > in the partitions method. The code demonstrates the problem: > {code:scala} > val wide = 20 > def unionRDD(num: Int): UnionRDD[Int] = { > val rdds = (0 until num).map(_ => sc.parallelize(1 to 10, 1)) > new UnionRDD(sc, rdds) > } > val level0 = (0 until wide).map { _ => > val level1 = (0 until wide).map(_ => unionRDD(wide)) > new UnionRDD(sc, level1) > } > val rdd = new UnionRDD(sc, level0) > rdd.partitions.length > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org