This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 97bf1ee9f6f7 [SPARK-47301][SQL][TESTS][FOLLOWUP] Remove workaround for
ParquetIOSuite
97bf1ee9f6f7 is described below
commit 97bf1ee9f6f76d49df50560bf792135308f289a9
Author: panbingkun <panbing...@baidu.com>
AuthorDate: Tue May 14 23:37:47 2024 +0800
[SPARK-47301][SQL][TESTS][FOLLOWUP] Remove workaround for ParquetIOSuite
### What changes were proposed in this pull request?
The pr aims to remove workaround for ParquetIOSuite.
### Why are the changes needed?
After https://github.com/apache/spark/pull/46562 is completed, the reason
why the ut `SPARK-7837 Do not close output writer twice when commitTask()
fails` failed due to different event processing time sequence no longer exists,
so we remove the previous workaround here.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Manually test.
- Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46577 from panbingkun/SPARK-47301_FOLLOWUP.
Authored-by: panbingkun <panbing...@baidu.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
.../spark/sql/execution/datasources/parquet/ParquetIOSuite.scala | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
index ba8fef0b3a8d..4fb8faa43a39 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
@@ -1589,12 +1589,8 @@ class ParquetIOWithoutOutputCommitCoordinationSuite
.coalesce(1)
df.write.partitionBy("a").options(extraOptions).parquet(dir.getCanonicalPath)
}
- if (m2.getErrorClass != null) {
- assert(m2.getErrorClass == "TASK_WRITE_FAILED")
- assert(m2.getCause.getMessage.contains("Intentional exception for
testing purposes"))
- } else {
- assert(m2.getMessage.contains("TASK_WRITE_FAILED"))
- }
+ assert(m2.getErrorClass == "TASK_WRITE_FAILED")
+ assert(m2.getCause.getMessage.contains("Intentional exception for
testing purposes"))
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
