This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 3bb0824 [SPARK-31853][DOCS] Mention removal of params mixins setter
in migration guide
3bb0824 is described below
commit 3bb08245a9c5e9f9da6f48e65e7dc0f80ad96b4a
Author: Enrico Minack <git...@enrico.minack.dev>
AuthorDate: Wed Jun 3 18:06:13 2020 -0500
[SPARK-31853][DOCS] Mention removal of params mixins setter in migration
guide
### What changes were proposed in this pull request?
The Pyspark Migration Guide needs to mention a breaking change of the
Pyspark ML API.
### Why are the changes needed?
In SPARK-29093, all setters have been removed from `Params` mixins in
`pyspark.ml.param.shared`. Those setters had been part of the public pyspark ML
API, hence this is a breaking change.
### Does this PR introduce _any_ user-facing change?
Only documentation.
### How was this patch tested?
Visually.
Closes #28663 from EnricoMi/branch-pyspark-migration-guide-setters.
Authored-by: Enrico Minack <git...@enrico.minack.dev>
Signed-off-by: Sean Owen <sro...@gmail.com>
(cherry picked from commit 4bbe3c2bb49030256d7e4f6941dd5629ee6d5b66)
Signed-off-by: Sean Owen <sro...@gmail.com>
---
docs/pyspark-migration-guide.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/pyspark-migration-guide.md b/docs/pyspark-migration-guide.md
index 6f0fbbf..2c9ea41 100644
--- a/docs/pyspark-migration-guide.md
+++ b/docs/pyspark-migration-guide.md
@@ -45,6 +45,8 @@ Please refer [Migration Guide: SQL, Datasets and
DataFrame](sql-migration-guide.
- As of Spark 3.0, `Row` field names are no longer sorted alphabetically when
constructing with named arguments for Python versions 3.6 and above, and the
order of fields will match that as entered. To enable sorted fields by default,
as in Spark 2.4, set the environment variable
`PYSPARK_ROW_FIELD_SORTING_ENABLED` to `true` for both executors and driver -
this environment variable must be consistent on all executors and driver;
otherwise, it may cause failures or incorrect answers. For [...]
+- In Spark 3.0, `pyspark.ml.param.shared.Has*` mixins do not provide any
`set*(self, value)` setter methods anymore, use the respective
`self.set(self.*, value)` instead. See
[SPARK-29093](https://issues.apache.org/jira/browse/SPARK-29093) for details.
+
## Upgrading from PySpark 2.3 to 2.4
- In PySpark, when Arrow optimization is enabled, previously `toPandas` just
failed when Arrow optimization is unable to be used whereas `createDataFrame`
from Pandas DataFrame allowed the fallback to non-optimization. Now, both
`toPandas` and `createDataFrame` from Pandas DataFrame allow the fallback by
default, which can be switched off by
`spark.sql.execution.arrow.fallback.enabled`.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
