Hi,
I might be asking something very trivial, but whats the recommend way of
using third party libraries.
I am using tables to read hdf5 format file..
And here is the error trace:
print rdd.take(2)
File "/tmp/spark/python/pyspark/rdd.py", line 1111, in take
res = self.context.runJob(self, takeUpToNumLeft, p, True)
File "/tmp/spark/python/pyspark/context.py", line 818, in runJob
it = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd,
javaPartitions, allowLocal)
File "/tmp/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 538, in __call__
File "/tmp/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line
300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage
0.0 (TID 3, srv-108-23.720.rdio):
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
File
"/hadoop/disk3/mapred/local/filecache/540/spark-assembly-1.2.0-hadoop2.3.0-cdh5.1.0.jar/pyspark/worker.py",
line 90, in main
command = pickleSer._read_with_length(infile)
File
"/hadoop/disk3/mapred/local/filecache/540/spark-assembly-1.2.0-hadoop2.3.0-cdh5.1.0.jar/pyspark/serializers.py",
line 151, in _read_with_length
return self.loads(obj)
File
"/hadoop/disk3/mapred/local/filecache/540/spark-assembly-1.2.0-hadoop2.3.0-cdh5.1.0.jar/pyspark/serializers.py",
line 396, in loads
return cPickle.loads(obj)
File
"/hadoop/disk3/mapred/local/filecache/540/spark-assembly-1.2.0-hadoop2.3.0-cdh5.1.0.jar/pyspark/cloudpickle.py",
line 825, in subimport
__import__(name)
ImportError: ('No module named tables', <function subimport at 0x47e1398>,
('tables',))
Though, import tables works fine on the local python shell.. but seems like
every thing is being pickled.. Are we expected to send all the files as
helper files? that doesn't seems right?
Thanks
--
Mohit
"When you want success as badly as you want the air, then you will get it.
There is no other secret of success."
-Socrates
