This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push: new a33d697e1e7 [SPARK-39273][PS][TESTS] Make PandasOnSparkTestCase inherit ReusedSQLTestCase a33d697e1e7 is described below commit a33d697e1e7c2854d77cc2302015ef54bf0c32ab Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Wed May 25 09:56:30 2022 +0900 [SPARK-39273][PS][TESTS] Make PandasOnSparkTestCase inherit ReusedSQLTestCase ### What changes were proposed in this pull request? This PR proposes to make `PandasOnSparkTestCase` inherit `ReusedSQLTestCase`. ### Why are the changes needed? We don't need this: ```python classmethod def tearDownClass(cls): # We don't stop Spark session to reuse across all tests. # The Spark session will be started and stopped at PyTest session level. # Please see pyspark/pandas/conftest.py. pass ``` anymore in Apache Spark. This has existed to speed up the tests when the codes are in Koalas repository where the tests run sequentially in single process. In Apache Spark, we run in multiple processes, and we don't need this anymore. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? Existing CI should test it out. Closes #36652 from HyukjinKwon/SPARK-39273. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> (cherry picked from commit a6dd6076d708713d11585bf7f3401d522ea48822) Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- python/pyspark/testing/pandasutils.py | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/python/pyspark/testing/pandasutils.py b/python/pyspark/testing/pandasutils.py index 9b07a23ae1b..baa43e5b9d5 100644 --- a/python/pyspark/testing/pandasutils.py +++ b/python/pyspark/testing/pandasutils.py @@ -18,7 +18,6 @@ import functools import shutil import tempfile -import unittest import warnings from contextlib import contextmanager from distutils.version import LooseVersion @@ -32,9 +31,8 @@ from pyspark import pandas as ps from pyspark.pandas.frame import DataFrame from pyspark.pandas.indexes import Index from pyspark.pandas.series import Series -from pyspark.pandas.utils import default_session, SPARK_CONF_ARROW_ENABLED -from pyspark.testing.sqlutils import SQLTestUtils - +from pyspark.pandas.utils import SPARK_CONF_ARROW_ENABLED +from pyspark.testing.sqlutils import ReusedSQLTestCase tabulate_requirement_message = None try: @@ -61,19 +59,12 @@ except ImportError as e: have_plotly = plotly_requirement_message is None -class PandasOnSparkTestCase(unittest.TestCase, SQLTestUtils): +class PandasOnSparkTestCase(ReusedSQLTestCase): @classmethod def setUpClass(cls): - cls.spark = default_session() + super(PandasOnSparkTestCase, cls).setUpClass() cls.spark.conf.set(SPARK_CONF_ARROW_ENABLED, True) - @classmethod - def tearDownClass(cls): - # We don't stop Spark session to reuse across all tests. - # The Spark session will be started and stopped at PyTest session level. - # Please see pyspark/pandas/conftest.py. - pass - def assertPandasEqual(self, left, right, check_exact=True): if isinstance(left, pd.DataFrame) and isinstance(right, pd.DataFrame): try: --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org