Yicong-Huang opened a new pull request, #58806: URL: https://github.com/apache/spark/pull/58806
### What changes were proposed in this pull request? This PR fixes outdated docstrings in `python/pyspark/serializers.py`. Three things are corrected: (1) The module docstring said PySpark uses `CloudPickleSerializer` to serialize objects using Python's `cPickle` serializer. `cPickle` is a Python 2 module that no longer exists in Python 3, and the actual default is the `CPickleSerializer` alias. The docstring now states that the default is `CPickleSerializer`, an alias that resolves to `CloudPickleSerializer` (backed by the `cloudpickle` library) by default and to `PickleSerializer` (backed by the standard-library `pickle` module) when `PYSPARK_ENABLE_NAMEDTUPLE_PATCH=1` is set, matching the alias logic in the code and the `SparkContext` serializer parameter docstring in `pyspark/core/context.py`. (2) The module docstring implied these serializers are what PySpark uses in general, but their scope is only the classic RDD path. A note is added clarifying that Spark SQL and DataFrame data transfers default to Arrow-based serialization since SPARK-54555, with a pointer to `pyspark.sql.pandas.serializers`. (3) Two documentation links pointing at the Python 2 docs (`http://docs.python.org/2/library/pickle.html` and `.../marshal.html`) are updated to the current `https://docs.python.org/3/...` URLs. ### Why are the changes needed? The docstrings are inaccurate and refer to Python 2 constructs (`cPickle`, `docs.python.org/2`) that no longer apply, which is misleading to readers of the PySpark source and API docs. ### Does this PR introduce _any_ user-facing change? No. Documentation-only change to docstrings. ### How was this patch tested? No behavior change. Only prose and documentation URLs in docstrings were edited; the doctest example blocks in the module docstring are unchanged, so existing doctest coverage still applies. Verified the file byte-compiles and that the module docstring still parses to the same set of doctest examples. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
