This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 7ce4dc64273 [SPARK-41775][PYTHON][FOLLOWUP] Use pyspark.cloudpickle instead of `cloudpickle` in torch distributor 7ce4dc64273 is described below commit 7ce4dc642736d78e79dff8e0b671cfd1b5d44166 Author: Weichen Xu <weichen...@databricks.com> AuthorDate: Sat May 27 09:06:39 2023 -0700 [SPARK-41775][PYTHON][FOLLOWUP] Use pyspark.cloudpickle instead of `cloudpickle` in torch distributor ### What changes were proposed in this pull request? Use pyspark.cloudpickle instead of `cloudpickle` in torch distributor ### Why are the changes needed? Make ser and deser code consistent ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Closes #41337 from WeichenXu123/fix-torch-distributor-import-cloudpickle. Authored-by: Weichen Xu <weichen...@databricks.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- python/pyspark/ml/torch/distributor.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/pyspark/ml/torch/distributor.py b/python/pyspark/ml/torch/distributor.py index 8a41bdcc886..ad8b4d8cc25 100644 --- a/python/pyspark/ml/torch/distributor.py +++ b/python/pyspark/ml/torch/distributor.py @@ -814,7 +814,7 @@ class TorchDistributor(Distributor): ) -> str: code = textwrap.dedent( f""" - import cloudpickle + from pyspark import cloudpickle import os if __name__ == "__main__": --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org