[ https://issues.apache.org/jira/browse/SPARK-32094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17145967#comment-17145967 ]
Hyukjin Kwon commented on SPARK-32094: -------------------------------------- Are you interested in submitting a PR to upgrade? > Patch cloudpickle.py with typing module side-effect fix > ------------------------------------------------------- > > Key: SPARK-32094 > URL: https://issues.apache.org/jira/browse/SPARK-32094 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.4.6, 3.0.0 > Reporter: Suzen Fylke > Priority: Major > > Pyspark's cloudpickle.py and versions of cloudpickle below 1.3.0 interfere > with dill unpickling because they define types.ClassType, which is undefined > in dill. This results in the following error: > {code:java} > Traceback (most recent call last): > File > "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", > line 279, in loads > return dill.loads(s) > File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 317, in > loads > return load(file, ignore) > File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 305, in > load > obj = pik.load() > File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 577, in > _load_type > return _reverse_typemap[name] > KeyError: 'ClassType'{code} > (See [https://github.com/cloudpipe/cloudpickle/issues/82]) > This was fixed for cloudpickle 1.3.0+ > ([https://github.com/cloudpipe/cloudpickle/pull/337]), but PySpark's > cloudpickle.py doesn't have this change yet. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org