Hi list,
I am having an issue ditributing a pandas_udf to my workers.
I'm using Spark 2.4.1 in standalone mode.
*Submit*:
- via SparkLauncher as separate process. I do add the py-files with the
self-executable zip (with .py extension) before launching the application.
- The whole application (few modules) is zippped into a single zip file
with 'py' extension
- The app defines a GROUPED_MAP pandas_udf that instanciate an class
from within the module
*Error*:
When I perform my df.gruopBy(<something>).agg(my_pandas_udf) my workers
errors out with:
====
2019-05-24 19:40:18.577: Caused by:
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
2019-05-24 19:40:18.577: File
"/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 366, in main
2019-05-24 19:40:18.577: func, profiler, deserializer, serializer =
read_udfs(pickleSer, infile, eval_type)
2019-05-24 19:40:18.577: File
"/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 229, in
read_udfs
2019-05-24 19:40:18.577: arg_offsets, udf = read_single_udf(pickleSer,
infile, eval_type, runner_conf)
2019-05-24 19:40:18.577: File
"/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 168, in
read_single_udf
2019-05-24 19:40:18.577: f, return_type = read_command(pickleSer,
infile)
2019-05-24 19:40:18.577: File
"/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 69, in
read_command
2019-05-24 19:40:18.577: command = serializer._read_with_length(file)
2019-05-24 19:40:18.577: File
"/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 172, in
_read_with_length
2019-05-24 19:40:18.577: return self.loads(obj)
2019-05-24 19:40:18.577: File
"/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 580, in
loads
2019-05-24 19:40:18.577: return pickle.loads(obj, encoding=encoding)
2019-05-24 19:40:18.577: ModuleNotFoundError: No module named '<my-module>'
2019-05-24 19:40:18.577:
2019-05-24 19:40:18.577: at
org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
====
where <my-module> is the name of my base package
*My python application:*
The structure is very simple
my-application.py (actually a zip):
- __mian__.py
- <my-module
- __init__.py
- <sub-package>
- __init__.py
- ...
- <another-sub-package>
- __init__
- spark_job.py
- udf.py
udf.py has a dummy class with just a staticmethod definig the UDF
spark_job.py: at some point defines the pandas_udf by calling:
pandas_udf(f=UDFHelper.function, returnType=UDFHelper.get_schema(),
functionType=PandasUDFType.GROUPED_MAP)
The error makes me think the executor don't have my archive in the
PYTHONPATH however the 'addPyFile' should take care of it.
I don't want to manually install my application everywhere (for env
constraints)
Any hint to troubleshoot or fix are really appreciated!
Thanks,