I get a serialization problem trying to run Python: sc.parallelize(['1','2']).map(lambda id: client.getRow('table', id, None))
cloudpickle.py can't pickle method_descriptor type I add a function to pickle a method descriptor and now it exceeds the recursion limit I print the method name before i pickle it and it is "reset" from cStringIO.StringO (output) The problem was at line ~830 of cloudpickle, trying to pickle a file And the initial object to pickle was that: (<function func at somewhere>, None, PairDeserializer(UTF8Deserializer(), UTF8Deserializer()), BatchedSerializer(PickleSerializer(), 0)) And the error is this: File "/home/user/inverted-index.py", line 80, in <module> print sc.wholeTextFiles(data_dir).flatMap(update).take(2)#.groupByKey().map(store).take(2) File "/home/user/spark2/python/pyspark/rdd.py", line 1081, in take totalParts = self._jrdd.partitions().size() File "/home/user/spark2/python/pyspark/rdd.py", line 2107, in _jrdd pickled_command = ser.dumps(command) File "/home/user/spark2/python/pyspark/serializers.py", line 402, in dumps return cloudpickle.dumps(obj, 2) File "/home/user/spark2/python/pyspark/cloudpickle.py", line 832, in dumps cp.dump(obj) File "/home/user/spark2/python/pyspark/cloudpickle.py", line 147, in dump raise pickle.PicklingError(msg) pickle.PicklingError: Could not pickle object as excessively deep recursion required. Try _fast_serialization=2 or contact PiCloud support Can any developer that works in that stuff tell me if that problem can be fixed? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/HBase-Thrift-API-Error-on-map-reduce-functions-tp21439.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org