[
https://issues.apache.org/jira/browse/SPARK-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288331#comment-14288331
]
Winston Chen edited comment on SPARK-5361 at 1/22/15 10:23 PM:
---------------------------------------------------------------
Found a good way to reproduce it:
{noformat}
from pyspark.rdd import RDD
dl = [
(u'2', {u'director': u'David Lean'}),
(u'7', {u'director': u'Andrew Dominik'})
]
dl_rdd = sc.parallelize(dl)
tmp = dl_rdd._to_java_object_rdd()
tmp2 = sc._jvm.SerDe.javaToPython(tmp)
t = RDD(tmp2, sc)
t.count()
tmp = t._to_java_object_rdd()
tmp2 = sc._jvm.SerDe.javaToPython(tmp)
t = RDD(tmp2, sc)
t.count() # it blows up here during the 2nd time of conversion
{noformat}
I am going to make a test case from this example.
was (Author: wingchen):
Found a good way to reproduce it:
{noformat}
from pyspark.rdd import RDD
dl = [
(u'2', {u'director': u'David Lean'}),
(u'7', {u'director': u'Andrew Dominik'})
]
dl_rdd = sc.parallelize(dl)
tmp = dl_rdd._to_java_object_rdd()
tmp2 = sc._jvm.SerDe.javaToPython(tmp)
t = RDD(tmp2, sc)
t.count()
tmp = t._to_java_object_rdd()
tmp2 = sc._jvm.SerDe.javaToPython(tmp)
t = RDD(tmp2, sc)
t.count() # it blows up here
{noformat}
I am going to make a test case from this example.
> python tuple not supported while converting PythonRDD back to JavaRDD
> ---------------------------------------------------------------------
>
> Key: SPARK-5361
> URL: https://issues.apache.org/jira/browse/SPARK-5361
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Reporter: Winston Chen
>
> Existing `SerDeUtil.pythonToJava` implementation does not count in tuple
> cases: Pyrolite `python tuple` => `java Object[]`.
> So with the following data:
> {noformat}
> [
> (u'2', {u'director': u'David Lean', u'genres': (u'Adventure', u'Biography',
> u'Drama'), u'title': u'Lawrence of Arabia', u'year': 1962}),
> (u'7', {u'director': u'Andrew Dominik', u'genres': (u'Biography', u'Crime',
> u'Drama'), u'title': u'The Assassination of Jesse James by the Coward Robert
> Ford', u'year': 2007})
> ]
> {noformat}
> Exceptions happen at the `genres` part:
> {noformat}
> 15/01/16 10:28:31 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 7)
> java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to
> java.util.ArrayList
> at
> org.apache.spark.api.python.SerDeUtil$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(SerDeUtil.scala:157)
> at
> org.apache.spark.api.python.SerDeUtil$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(SerDeUtil.scala:153)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)
> {noformat}
> There is already a pull-request for this bug:
> https://github.com/apache/spark/pull/4146
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
