[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117252#comment-17117252 ]
Apache Spark commented on SPARK-31788: -------------------------------------- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/28648 > Error when creating UnionRDD of PairRDDs > ---------------------------------------- > > Key: SPARK-31788 > URL: https://issues.apache.org/jira/browse/SPARK-31788 > Project: Spark > Issue Type: Bug > Components: DStreams, PySpark, Spark Core > Affects Versions: 3.0.0 > Reporter: Sanket Reddy > Assignee: Hyukjin Kwon > Priority: Blocker > > Union RDD of Pair RDD's seems to have issues > SparkSession available as 'spark'. > {code} > rdd1 = sc.parallelize([1,2,3,4,5]) > rdd2 = sc.parallelize([6,7,8,9,10]) > pairRDD1 = rdd1.zip(rdd2) > unionRDD1 = sc.union([pairRDD1, pairRDD1]) > {code} > {code} > Traceback (most recent call last): File "<stdin>", line 1, in <module> File > "/home/gs/spark/latest/python/pyspark/context.py", line 870, > in union jrdds[i] = rdds[i]._jrdd > File > "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", > line 238, in _setitem_ File > "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", > line 221, > in __set_item File > "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line > 332, in get_return_value py4j.protocol.Py4JError: An error occurred while > calling None.None. Trace: py4j.Py4JException: Cannot convert > org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD at > py4j.commands.ArrayCommand.convertArgument(ArrayCommand.java:166) at > py4j.commands.ArrayCommand.setArray(ArrayCommand.java:144) at > py4j.commands.ArrayCommand.execute(ArrayCommand.java:97) at > py4j.GatewayConnection.run(GatewayConnection.java:238) at > java.lang.Thread.run(Thread.java:748) > {code} > {code} > rdd3 = sc.parallelize([11,12,13,14,15]) > pairRDD2 = rdd3.zip(rdd3) > unionRDD2 = sc.union([pairRDD1, pairRDD2]) > {code} > {code} > Traceback (most recent call last): File "<stdin>", line 1, in <module> File > "/home/gs/spark/latest/python/pyspark/context.py", line 870, in union > jrdds[i] = rdds[i]._jrdd File > "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", > line 238, in _setitem_ File > "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", > line 221, in __set_item File > "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line > 332, in get_return_value py4j.protocol.Py4JError: An error occurred while > calling None.None. Trace: py4j.Py4JException: Cannot convert > org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD at > py4j.commands.ArrayCommand.convertArgument(ArrayCommand.java:166) at > py4j.commands.ArrayCommand.setArray(ArrayCommand.java:144) at > py4j.commands.ArrayCommand.execute(ArrayCommand.java:97) at > py4j.GatewayConnection.run(GatewayConnection.java:238) at > java.lang.Thread.run(Thread.java:748) > {code} > 2.4.5 does not have this regression as below: > {code} > rdd4 = sc.parallelize(range(5)) > pairRDD3 = rdd4.zip(rdd4) > unionRDD3 = sc.union([pairRDD1, pairRDD3]) > unionRDD3.collect() > {code} > {code} > [(1, 6), (2, 7), (3, 8), (4, 9), (5, 10), (0, 0), (1, 1), (2, 2), (3, 3), (4, > 4)] > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org