GitHub user superbobry opened a pull request:
https://github.com/apache/spark/pull/21157
[SPARK-22674][PYTHON] Removed the namedtuple pickling patch
## What changes were proposed in this pull request?
This is a breaking change.
Prior to this commit PySpark patched ``collections.namedtuple`` to make
namedtuple instances serializable even if the namedtuple class has been
defined outside of ``globals()``, e.g.
def do_something():
Foo = namedtuple("Foo", ["foo"])
sc.parallelize(range(1)).map(lambda _: Foo(42))
The patch changed the pickled representation of the namedtuple instance
to include the structure of namedtuple class, and recreate the class on
each unpickling. This behaviour causes hard to diagnose failures both
in the user code with namedtuples, as well as third-party libraries
relying on them. See [1] and [2] for details.
[1]:
https://superbobry.github.io/pyspark-silently-breaks-your-namedtuples.html
[2]:
https://superbobry.github.io/tensorflowonspark-or-the-namedtuple-patch-strikes-again.html
## How was this patch tested?
PySpark test suite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/criteo-forks/spark no-hijack-namedtuple
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21157.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21157
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
