This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 17d283990b6 [SPARK-45553][PS] Deprecate `assertPandasOnSparkEqual`
17d283990b6 is described below
commit 17d283990b64614828838afa718f48b855ab7842
Author: Haejoon Lee <haejoon....@databricks.com>
AuthorDate: Thu Oct 19 13:57:01 2023 +0900
[SPARK-45553][PS] Deprecate `assertPandasOnSparkEqual`
### What changes were proposed in this pull request?
This PR proposes to deprecate `assertPandasOnSparkEqual`.
### Why are the changes needed?
Now we have more pandas friendly testing utils such as
`ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal` and
`ps.testing.assert_index_equal`.
### Does this PR introduce _any_ user-facing change?
Not for now, but `assertPandasOnSparkEqual` will be removed in the future
version.
### How was this patch tested?
The existing CI should pass.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43426 from itholic/SPARK-45553.
Authored-by: Haejoon Lee <haejoon....@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit f3e280b952da8b8ab6c78371f3715cc674a73bc1)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/testing/pandasutils.py | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/python/pyspark/testing/pandasutils.py
b/python/pyspark/testing/pandasutils.py
index c80ffb7ee53..04a523bce76 100644
--- a/python/pyspark/testing/pandasutils.py
+++ b/python/pyspark/testing/pandasutils.py
@@ -365,6 +365,11 @@ def assertPandasOnSparkEqual(
.. versionadded:: 3.5.0
+ .. deprecated:: 3.5.1
+ `assertPandasOnSparkEqual` will be removed in Spark 4.0.0.
+ Use `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal`
+ and `ps.testing.assert_index_equal` instead.
+
Parameters
----------
actual: pandas-on-Spark DataFrame, Series, or Index
@@ -417,6 +422,12 @@ def assertPandasOnSparkEqual(
>>> s2 = ps.Index([212.3, 100.0001])
>>> assertPandasOnSparkEqual(s1, s2, almost=True) # pass, ps.Index obj
are almost equal
"""
+ warnings.warn(
+ "`assertPandasOnSparkEqual` will be removed in Spark 4.0.0. "
+ "Use `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal`
"
+ "and `ps.testing.assert_index_equal` instead.",
+ FutureWarning,
+ )
if actual is None and expected is None:
return True
elif actual is None or expected is None:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
