[ https://issues.apache.org/jira/browse/SPARK-17756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reassigned SPARK-17756: ------------------------------------ Assignee: Hyukjin Kwon > java.lang.ClassCastException when using cartesian with DStream.transform > ------------------------------------------------------------------------ > > Key: SPARK-17756 > URL: https://issues.apache.org/jira/browse/SPARK-17756 > Project: Spark > Issue Type: Bug > Components: DStreams, PySpark > Affects Versions: 2.0.0, 2.3.0 > Reporter: Maciej Szymkiewicz > Assignee: Hyukjin Kwon > Priority: Major > Fix For: 2.4.0 > > > Steps to reproduce: > {code} > from pyspark.streaming import StreamingContext > ssc = StreamingContext(spark.sparkContext, 10) > (ssc > .queueStream([sc.range(10)]) > .transform(lambda rdd: rdd.cartesian(rdd)) > .pprint()) > ssc.start() > ## 16/10/01 21:34:30 ERROR JobScheduler: Error generating jobs for time > 1475350470000 ms > ## java.lang.ClassCastException: org.apache.spark.api.java.JavaPairRDD ## > cannot be cast to org.apache.spark.api.java.JavaRDD > ## at com.sun.proxy.$Proxy15.call(Unknown Source) > ## .... > {code} > A dummy fix is to put {{map(lamba x: x)}} which suggests it is a problem > similar to https://issues.apache.org/jira/browse/SPARK-16589 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org