Thanks Davies and Eric. I followed Davies' instructions and it works
wonderful.
I would add that you can also add these scripts in the pyspark shell too:
pyspark --py-files support.py
where support.py is your script containing your class as Davies described.
Best,
Guillaume Guy
* +1 919 -
Hi,
This is a duplicate of the stack-overflow question here
http://stackoverflow.com/questions/28569374/spark-returning-pickle-error-cannot-lookup-attribute.
I hope to generate more interest on this mailing list.
*The problem:*
I am running into some attribute lookup problems when trying to
Currently, PySpark can not support pickle a class object in current
script ( '__main__'), the workaround could be put the implementation
of the class into a separate module, then use bin/spark-submit
--py-files xxx.py in deploy it.
in xxx.py:
class test(object):
def __init__(self, a, b):
