This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 997100119e6c [SPARK-48083][SPARK-48084][ML][TESTS] Remove JIRA
comments for reenabling ML compatibility tests
997100119e6c is described below
commit 997100119e6c893c46b12cc32a4f96721c8f3a22
Author: Weichen Xu <weichen...@databricks.com>
AuthorDate: Tue May 7 15:58:01 2024 +0900
[SPARK-48083][SPARK-48084][ML][TESTS] Remove JIRA comments for reenabling
ML compatibility tests
### What changes were proposed in this pull request?
Enable spark ml test
### Why are the changes needed?
For test_connect_classification.py,
session.copyFromLocalToFs failure with 3.5 client <> 4.0 server
this is not an issue,
copyFromLocalToFs requires spark server to config
spark.connect.copyFromLocalToFs.allowDestLocal to False, because the test can
only use local fs.
For test_connect_evaluation.py,
This test error pyspark.ml.connect.evaluation not working in 3.5 client <>
4.0 server is caused by cloudpickle forward incompatibility, it is not related
to ML code
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46419 from WeichenXu123/reenable-test-3.5.
Authored-by: Weichen Xu <weichen...@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/ml/tests/connect/test_connect_classification.py | 1 -
python/pyspark/ml/tests/connect/test_connect_evaluation.py | 1 -
2 files changed, 2 deletions(-)
diff --git a/python/pyspark/ml/tests/connect/test_connect_classification.py
b/python/pyspark/ml/tests/connect/test_connect_classification.py
index 2763d3f613ae..f0d60a117e12 100644
--- a/python/pyspark/ml/tests/connect/test_connect_classification.py
+++ b/python/pyspark/ml/tests/connect/test_connect_classification.py
@@ -21,7 +21,6 @@ import os
from pyspark.sql import SparkSession
from pyspark.ml.tests.connect.test_legacy_mode_classification import
ClassificationTestsMixin
-# TODO(SPARK-48083): Reenable this test case
have_torch = "SPARK_SKIP_CONNECT_COMPAT_TESTS" not in os.environ
try:
import torch # noqa: F401
diff --git a/python/pyspark/ml/tests/connect/test_connect_evaluation.py
b/python/pyspark/ml/tests/connect/test_connect_evaluation.py
index 35af54605ca8..58076dfe0bbe 100644
--- a/python/pyspark/ml/tests/connect/test_connect_evaluation.py
+++ b/python/pyspark/ml/tests/connect/test_connect_evaluation.py
@@ -20,7 +20,6 @@ import unittest
from pyspark.sql import SparkSession
from pyspark.ml.tests.connect.test_legacy_mode_evaluation import
EvaluationTestsMixin
-# TODO(SPARK-48084): Reenable this test case
have_torcheval = "SPARK_SKIP_CONNECT_COMPAT_TESTS" not in os.environ
try:
import torcheval # noqa: F401
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
