[
https://issues.apache.org/jira/browse/SPARK-23159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-23159.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 20373
[https://github.com/apache/spark/pull/20373]
> Update Cloudpickle to match version 0.4.3
> -----------------------------------------
>
> Key: SPARK-23159
> URL: https://issues.apache.org/jira/browse/SPARK-23159
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.3.0
> Reporter: Bryan Cutler
> Assignee: Bryan Cutler
> Priority: Major
> Fix For: 2.4.0
>
>
> Update PySpark's version of Cloudpickle to match version 0.4.3. The reasons
> for doing this are:
> * Pick up bug fixes, improvements with newer version
> * Match a specific version as close as possible (Spark has additional
> changes that might be necessary) to make future upgrades easier
> There are newer versions of Cloudpickle that can fix bugs with NamedTuple
> pickling (that Spark currently has workarounds for), but these include other
> changes that need some verification before bringing into Spark. Upgrading
> first to 0.4.3 will help make this verification easier.
> Discussion on the mailing list:
> [http://apache-spark-developers-list.1001551.n3.nabble.com/Thoughts-on-Cloudpickle-Update-td23188.html]
> Upgrading to the recent release of v0.4.3 will include the following:
> * Fix pickling of named tuples
> [https://github.com/cloudpipe/cloudpickle/pull/113]
> * Built in type constructors for PyPy compatibility
> [here]([https://github.com/cloudpipe/cloudpickle/commit/d84980ccaafc7982a50d4e04064011f401f17d1b])
> * Fix memoryview support [https://github.com/cloudpipe/cloudpickle/pull/122]
> * Improved compatibility with other cloudpickle versions
> [https://github.com/cloudpipe/cloudpickle/pull/128]
> * Several cleanups [https://github.com/cloudpipe/cloudpickle/pull/121] and
> [here]([https://github.com/cloudpipe/cloudpickle/commit/c91aaf110441991307f5097f950764079d0f9652])
> * [MRG] Regression on pickling classes from the __main__ module
> [https://github.com/cloudpipe/cloudpickle/pull/149]
> * BUG: Handle instance methods of builtin types
> [https://github.com/cloudpipe/cloudpickle/pull/154]
> * Fix #129 : do not silence RuntimeError in dump()
> [https://github.com/cloudpipe/cloudpickle/pull/153]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
