This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new e298a136881 [SPARK-39425][PYTHON][PS] Add migration guide for pandas 1.4 behavior changes e298a136881 is described below commit e298a136881f734db24f339b80341d8ff888bacc Author: Yikun Jiang <yikunk...@gmail.com> AuthorDate: Fri Jun 10 17:22:51 2022 +0900 [SPARK-39425][PYTHON][PS] Add migration guide for pandas 1.4 behavior changes ### What changes were proposed in this pull request? Add migration guide for pandas 1.4 behavior changes: * SPARK-39054 https://github.com/apache/spark/pull/36581: In Spark 3.4, if Pandas on Spark API `Groupby.apply`'s `func` parameter return type is not specified and `compute.shortcut_limit` is set to `0`, the sampling rows will be set to 2 (ensure sampling rows always >= 2) to make sure infer schema is accurate. * SPARK-38822 https://github.com/apache/spark/pull/36168 In Spark 3.4, if Pandas on Spark API `Index.insert` is out of bounds, will rasied IndexError with `index {} is out of bounds for axis 0 with size {}` to follow pandas 1.4 behavior. * SPARK-38857 https://github.com/apache/spark/pull/36159 In Spark 3.4, the series name will be preserved in Pandas on Spark API `Series.mode` to follow pandas 1.4 behavior. * SPARK-38859 https://github.com/apache/spark/pull/36142 In Spark 3.4, the Pandas on Spark API `Index.__setitem__` will first to check `value` type is `Column` to avoid raise unexpected error in `is_list_like` like `Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.`. * SPARK-38820 https://github.com/apache/spark/pull/36357 In Spark 3.4, the Pandas on Spark API `astype('category')` will also refresh `categories.dtype` according to original data `dtype` to follow pandas 1.4 behavior. * SPARK-38947 https://github.com/apache/spark/pull/36464 In Spark 3.4, the Pandas on Spark API supports groupby positional indexing in `GroupBy.head` and `GroupBy.tail` to follow pandas 1.4. Negative arguments now work correctly and result in ranges relative to the end and start of each group, Previously, negative arguments returned empty frames. * SPARK-39317 https://github.com/apache/spark/pull/36699 In Spark 3.4, the infer schema process of `groupby.apply` in Pandas on Spark, will first infer the pandas type to ensure the accuracy of the pandas `dtype` as much as possible. * SPARK-39314 https://github.com/apache/spark/pull/36711 In Spark 3.4, the `Series.concat` sort parameter will be respected to follow pandas 1.4 behaviors. For other test only fixes, I don't add migration doc: SPARK-38821 SPARK-39053 SPARK-38982 ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes #36816 from Yikun/SPARK-39425. Authored-by: Yikun Jiang <yikunk...@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .../docs/source/migration_guide/pyspark_3.3_to_3.4.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst b/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst index 9f8cf545e28..dbe7b818b2a 100644 --- a/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst +++ b/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst @@ -20,4 +20,20 @@ Upgrading from PySpark 3.3 to 3.4 ================================= -* In Spark 3.4, the schema of an array column is inferred by merging the schemas of all elements in the array. To restore the previous behavior where the schema is only inferred from the first element, you can set ``spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled`` to ``true``. \ No newline at end of file +* In Spark 3.4, the schema of an array column is inferred by merging the schemas of all elements in the array. To restore the previous behavior where the schema is only inferred from the first element, you can set ``spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled`` to ``true``. + +* In Spark 3.4, if Pandas on Spark API ``Groupby.apply``'s ``func`` parameter return type is not specified and ``compute.shortcut_limit`` is set to 0, the sampling rows will be set to 2 (ensure sampling rows always >= 2) to make sure infer schema is accurate. + +* In Spark 3.4, if Pandas on Spark API ``Index.insert`` is out of bounds, will raise IndexError with ``index {} is out of bounds for axis 0 with size {}`` to follow pandas 1.4 behavior. + +* In Spark 3.4, the series name will be preserved in Pandas on Spark API ``Series.mode`` to follow pandas 1.4 behavior. + +* In Spark 3.4, the Pandas on Spark API ``Index.__setitem__`` will first to check ``value`` type is ``Column`` type to avoid raising unexpected ``ValueError`` in ``is_list_like`` like `Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.`. + +* In Spark 3.4, the Pandas on Spark API ``astype('category')`` will also refresh ``categories.dtype`` according to original data ``dtype`` to follow pandas 1.4 behavior. + +* In Spark 3.4, the Pandas on Spark API supports groupby positional indexing in ``GroupBy.head`` and ``GroupBy.tail`` to follow pandas 1.4. Negative arguments now work correctly and result in ranges relative to the end and start of each group, Previously, negative arguments returned empty frames. + +* In Spark 3.4, the infer schema process of ``groupby.apply`` in Pandas on Spark, will first infer the pandas type to ensure the accuracy of the pandas ``dtype`` as much as possible. + +* In Spark 3.4, the ``Series.concat`` sort parameter will be respected to follow pandas 1.4 behaviors. --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org