This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new b6656afa4e5 [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import `WriteOperation` only when `pandas` is available b6656afa4e5 is described below commit b6656afa4e54e489c4642522ec74d850f53d3a83 Author: Dongjoon Hyun <dongj...@apache.org> AuthorDate: Tue Dec 6 03:03:17 2022 -0800 [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import `WriteOperation` only when `pandas` is available ### What changes were proposed in this pull request? This is the last piece to recover `pyspark-connect` tests on a system where pandas is unavailable. ### Why are the changes needed? **BEFORE** ``` $ python/run-tests.py --modules pyspark-connect ... ModuleNotFoundError: No module named 'pandas' ``` **AFTER** ``` $ python/run-tests.py --modules pyspark-connect Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark-merge/python/unit-tests.log Will test against the following Python executables: ['python3.9'] Will test the following Python modules: ['pyspark-connect'] python3.9 python_implementation is CPython python3.9 version is: Python 3.9.14 Starting test(python3.9): pyspark.sql.tests.connect.test_connect_column_expressions (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/fc8f073e-edbe-470f-a3cc-f35b3f4b5262/python3.9__pyspark.sql.tests.connect.test_connect_column_expressions___lzz5fky.log) Starting test(python3.9): pyspark.sql.tests.connect.test_connect_basic (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/c14dfac4-14df-45b4-b54e-6dbc3c58f128/python3.9__pyspark.sql.tests.connect.test_connect_basic__x8rz2dz1.log) Starting test(python3.9): pyspark.sql.tests.connect.test_connect_column (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/5810e639-75c1-4e6c-91b3-2a1894dab319/python3.9__pyspark.sql.tests.connect.test_connect_column__3e_qmue7.log) Starting test(python3.9): pyspark.sql.tests.connect.test_connect_function (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/8bebb3be-c146-4030-96b3-82ff7a873d49/python3.9__pyspark.sql.tests.connect.test_connect_function__hwe9aca8.log) Finished test(python3.9): pyspark.sql.tests.connect.test_connect_column_expressions (1s) ... 9 tests were skipped Starting test(python3.9): pyspark.sql.tests.connect.test_connect_plan_only (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/97cbccca-1e9a-44d9-958f-e2368266b437/python3.9__pyspark.sql.tests.connect.test_connect_plan_only__dffs5ux1.log) Finished test(python3.9): pyspark.sql.tests.connect.test_connect_column (1s) ... 2 tests were skipped Starting test(python3.9): pyspark.sql.tests.connect.test_connect_select_ops (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/297a054f-c577-4aee-b1ff-48d48beb423c/python3.9__pyspark.sql.tests.connect.test_connect_select_ops__4e__w6gh.log) Finished test(python3.9): pyspark.sql.tests.connect.test_connect_function (1s) ... 5 tests were skipped Finished test(python3.9): pyspark.sql.tests.connect.test_connect_basic (1s) ... 48 tests were skipped Finished test(python3.9): pyspark.sql.tests.connect.test_connect_select_ops (0s) ... 2 tests were skipped Finished test(python3.9): pyspark.sql.tests.connect.test_connect_plan_only (1s) ... 28 tests were skipped Tests passed in 2 seconds Skipped tests in pyspark.sql.tests.connect.test_connect_basic with python3.9: test_channel_properties (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.002s) test_invalid_connection_strings (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s) test_metadata (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s) test_sensible_defaults (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s) test_valid_channel_creation (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s) test_agg_with_avg (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_agg_with_two_agg_exprs (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_collect (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_count (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_create_global_temp_view (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_create_session_local_temp_view (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_crossjoin (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_deduplicate (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_drop (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_drop_na (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_empty_dataset (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_explain_string (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_fill_na (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_first (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_head (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_input_files (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_is_empty (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_is_local (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_is_streaming (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_join_condition_column_list_columns (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_limit_offset (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_print_schema (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_range (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_replace (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_repr (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_schema (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_select_expr (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_session (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_show (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_simple_datasource_read (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_simple_explain_string (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_simple_read (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_simple_udf (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_sort (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) test_sql (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_subquery_alias (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_tail (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_take (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_toDF (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_to_pandas (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_with_columns (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_with_columns_renamed (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s) test_write_operations (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s) Skipped tests in pyspark.sql.tests.connect.test_connect_column with python3.9: test_column_operator (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests) ... skip (0.002s) test_columns (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests) ... skip (0.001s) Skipped tests in pyspark.sql.tests.connect.test_connect_column_expressions with python3.9: test_binary_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.002s) test_column_alias (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) test_column_literals (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) test_float_nan_inf (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) test_map_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.001s) test_null_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) test_numeric_literal_types (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) test_simple_column_expressions (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) test_uuid_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s) Skipped tests in pyspark.sql.tests.connect.test_connect_function with python3.9: test_aggregation_functions (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.006s) test_math_functions (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.025s) test_normal_functions (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.002s) test_sort_with_nulls_order (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.001s) test_sorting_functions_with_column (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.000s) Skipped tests in pyspark.sql.tests.connect.test_connect_plan_only with python3.9: test_all_the_plans (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.013s) test_coalesce_and_repartition (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.002s) test_crossjoin (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_crosstab (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_datasource_read (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_deduplicate (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_drop (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_drop_na (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_except (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_fill_na (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_filter (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_intersect (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_join_condition (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_join_using_columns (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_limit (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_offset (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_range (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_relation_alias (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_replace (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_sample (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_simple_project (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_simple_udf (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s) test_sort (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_sql_project (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.002s) test_summary (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_union (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.002s) test_unsupported_functions (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) test_write_operation (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s) Skipped tests in pyspark.sql.tests.connect.test_connect_select_ops with python3.9: test_join_with_join_type (pyspark.sql.tests.connect.test_connect_select_ops.SparkConnectToProtoSuite) ... skip (0.085s) test_select_with_columns_and_strings (pyspark.sql.tests.connect.test_connect_select_ops.SparkConnectToProtoSuite) ... skip (0.000s) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass the CIs. Closes #38934 from dongjoon-hyun/SPARK-41317. Authored-by: Dongjoon Hyun <dongj...@apache.org> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- python/pyspark/sql/tests/connect/test_connect_plan_only.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/pyspark/sql/tests/connect/test_connect_plan_only.py b/python/pyspark/sql/tests/connect/test_connect_plan_only.py index 590c90aef97..0d52e66afd0 100644 --- a/python/pyspark/sql/tests/connect/test_connect_plan_only.py +++ b/python/pyspark/sql/tests/connect/test_connect_plan_only.py @@ -17,13 +17,13 @@ from typing import cast import unittest -from pyspark.sql.connect.plan import WriteOperation from pyspark.testing.connectutils import PlanOnlyTestFixture from pyspark.testing.sqlutils import have_pandas, pandas_requirement_message if have_pandas: import pyspark.sql.connect.proto as proto from pyspark.sql.connect.column import Column + from pyspark.sql.connect.plan import WriteOperation from pyspark.sql.connect.readwriter import DataFrameReader from pyspark.sql.connect.function_builder import UserDefinedFunction, udf from pyspark.sql.types import StringType --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org