This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 60bf6817d862 [SPARK-46262][PS] Enable test for `np.left_shift` for Pandas-on-Spark object 60bf6817d862 is described below commit 60bf6817d862a8daac629156280996f7dfd07ba7 Author: Haejoon Lee <haejoon....@databricks.com> AuthorDate: Tue Dec 5 18:33:04 2023 +0800 [SPARK-46262][PS] Enable test for `np.left_shift` for Pandas-on-Spark object ### What changes were proposed in this pull request? This PR proposes to enable test for `np.left_shift` for Pandas-on-Spark object. ### Why are the changes needed? Because now we completely support the PyArrow>=4.0.0, so the behavior difference within PyArrow<0.15 will not appear. So we can enable the test since the behavior always same within the supported version as below: ```python >>> pdf = pd.DataFrame({"A": [1, 2, 3, 4]}) >>> psdf = ps.DataFrame({"A": [1, 2, 3, 4]}) >>> np.left_shift(pdf, 1) A 0 2 1 4 2 6 3 8 >>> np.left_shift(psdf, 1) A 0 2 1 4 2 6 3 8 ``` ### Does this PR introduce _any_ user-facing change? No, this just enables the test case. ### How was this patch tested? The existing CI especially `NumPyCompatTests` should pass. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44178 from itholic/enable_np_left_shift. Authored-by: Haejoon Lee <haejoon....@databricks.com> Signed-off-by: Ruifeng Zheng <ruife...@apache.org> --- python/pyspark/pandas/tests/test_numpy_compat.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py index af3dcdf9f71f..986083d0d3af 100644 --- a/python/pyspark/pandas/tests/test_numpy_compat.py +++ b/python/pyspark/pandas/tests/test_numpy_compat.py @@ -41,10 +41,6 @@ class NumPyCompatTestsMixin: "log1p", # flaky "modf", "floor_divide", # flaky - # Results seem inconsistent in a different version of, I (Hyukjin) suspect, PyArrow. - # From PyArrow 0.15, seems it returns the correct results via PySpark. Probably we - # can enable it later when Koalas switches to PyArrow 0.15 completely. - "left_shift", ] @property --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org