I get a serialization problem trying to run
Python:
sc.parallelize(['1','2']).map(lambda id: client.getRow('table', id, None))
cloudpickle.py can't pickle method_descriptor type
I add a function to pickle a method descriptor and now it exceeds the
recursion limit
I print the method name before i pickle it and it is "reset" from
cStringIO.StringO (output)
The problem was at line ~830 of cloudpickle, trying to pickle a file
And the initial object to pickle was that:
(<function func at somewhere>, None, PairDeserializer(UTF8Deserializer(),
UTF8Deserializer()), BatchedSerializer(PickleSerializer(), 0))
And the error is this:
File "/home/user/inverted-index.py", line 80, in <module>
print
sc.wholeTextFiles(data_dir).flatMap(update).take(2)#.groupByKey().map(store).take(2)
File "/home/user/spark2/python/pyspark/rdd.py", line 1081, in take
totalParts = self._jrdd.partitions().size()
File "/home/user/spark2/python/pyspark/rdd.py", line 2107, in _jrdd
pickled_command = ser.dumps(command)
File "/home/user/spark2/python/pyspark/serializers.py", line 402, in dumps
return cloudpickle.dumps(obj, 2)
File "/home/user/spark2/python/pyspark/cloudpickle.py", line 832, in dumps
cp.dump(obj)
File "/home/user/spark2/python/pyspark/cloudpickle.py", line 147, in dump
raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion
required.
Try _fast_serialization=2 or contact PiCloud support
Can any developer that works in that stuff tell me if that problem can be
fixed?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/HBase-Thrift-API-Error-on-map-reduce-functions-tp21439.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
