HyukjinKwon opened a new pull request, #60120: URL: https://github.com/apache/spark/pull/60120
### What changes were proposed in this pull request? This is a POC that replaces the copy of cloudpickle 3.1.2 bundled as `pyspark.cloudpickle` with [cloudpickle-rust](https://github.com/HyukjinKwon/cloudpickle-rust) 0.1.0, a Rust (PyO3) implementation of cloudpickle with the same API, producing the same payloads except for the module of the reconstructors (`cloudpickle_rust.cloudpickle`). - `pyspark.cloudpickle` stays as a thin module on top of `cloudpickle_rust`, so PySpark itself and third-party code keep importing it as before: - `pyspark.cloudpickle.cloudpickle` and `pyspark.cloudpickle.cloudpickle_fast` are aliases of the `cloudpickle_rust` modules. Payloads pickled by previous PySpark versions reference `pyspark.cloudpickle.cloudpickle.<reconstructor>`, for example the UDFs sent by an older Spark Connect client (the `pyspark-connect-old-client` job), so they still unpickle. - Its `Pickler` keeps the original `co_filename` of the code pickled by value. cloudpickle-rust, like upstream cloudpickle master ([cloudpipe/cloudpickle#560](https://github.com/cloudpipe/cloudpickle/pull/560), not released yet), replaces it with `"<dynamic-code>"`. That would remove the file names from the tracebacks of Python UDFs, break the perf profiler output, and make the memory profiler fail with `MEMORY_PROFILE_INVALID_SOURCE` (20 tests in `test_udf_profiler` failed without it). The override is a dispatch table of the same native type as cloudpickle-rust's, so only code objects go through Python. - `cloudpickle-rust>=0.1.0` becomes a required dependency of `pyspark` and `pyspark-client`. It is added to the `pyproject.toml` dependency groups that the CI images install, to `dev/requirements.txt`, to the workflows that pip install packages directly (Scala tests that run Python UDFs, Maven, macOS, the Spark 4.0 client against the master server), to the release image, and to the Kubernetes PySpark image. - Removes the bundled source and its license references (`LICENSE`, `LICENSE-binary`, `licenses*/LICENSE-cloudpickle.txt`), and its ruff, mypy, coverage and RAT exclusions. - Documents the new dependency in the installation guide and the in-process UDF guide. ### Why are the changes needed? To see if cloudpickle-rust can replace the bundled cloudpickle. It is 1.4x to 1.7x faster to pickle dynamic functions, classes and their instances (see [its benchmarks](https://github.com/HyukjinKwon/cloudpickle-rust#performance)), passes the upstream cloudpickle test suite with the same results, and supports free-threaded CPython without re-enabling the GIL. ### Does this PR introduce _any_ user-facing change? Yes. PySpark requires cloudpickle-rust, which is a native extension. The bundled copy is imported by executors from `pyspark.zip`, but cloudpickle-rust has to be installed in the Python environments of the driver and of every executor (cluster images, `--archives` virtual environments, and so on). The Python functions and classes pickled by value, such as UDFs and RDD closures, now reference `cloudpickle_rust.cloudpickle`, so they cannot be unpickled where cloudpickle-rust is not installed, for example by a Spark Connect server of a version without this change. ### How was this patch tested? By the CI of this PR, which runs all the tests because `pyproject.toml` changed. Locally on macOS arm64 with CPython 3.14: - `pyspark.tests.test_serializers`, `test_worker`, `test_rdd`, `test_broadcast`, `test_profiler`, `test_memory_profiler`, `pyspark.sql.tests.test_udf_profiler`, and the doctests of `pyspark.serializers` and `pyspark.accumulators`. - Payloads pickled by the bundled cloudpickle of master (3.1.2) and branch-4.0 (3.1.1) unpickle with this change and pickle again: lambdas, closures, a UDF command, dynamic classes and instances, enums, dataclasses, `TypeVar`s, namedtuples, dict views and module references. - `CloudPickler` subclasses such as `_InProcessPickler` still get their `reducer_override` called. - mypy with the invocation of `dev/lint-python`, and ruff. Not covered yet, and needed before this could go in: - Python 3.15: cloudpickle-rust 0.1.0 requires Python < 3.15, so the scheduled Python 3.15 build cannot install it. - The target version, a migration guide entry, and the official Docker images (apache/spark-docker). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
