LuciferYang commented on code in PR #12981:
URL: https://github.com/apache/gluten/pull/12981#discussion_r3964889807
##########
backends-clickhouse/src/test/scala/org/apache/gluten/execution/GlutenClickHouseWholeStageTransformerSuite.scala:
##########
@@ -155,15 +154,16 @@ class GlutenClickHouseWholeStageTransformerSuite
final override protected val resourcePath: String = "" // ch not need this
override protected val fileFormat: String = "parquet"
- protected def testSparkVersionLE33(testName: String, testTag: Tag*)(testFun:
=> Any): Unit = {
- if (isSparkVersionLE("3.3")) {
- test(testName, testTag: _*)(testFun)
- } else {
- ignore(s"[$SPARK_VERSION_SHORT]-$testName", testTag: _*)(testFun)
- }
+ /**
+ * These cases only ever passed on Spark 3.3, which is no longer supported.
They are registered as
+ * ignored rather than deleted so that they stay visible in the test report
until someone either
+ * makes them pass on a supported version or removes them.
+ */
+ protected def ignoreSpark33OnlyCase(testName: String, testTag:
Tag*)(testFun: => Any): Unit = {
Review Comment:
These 13 call sites are ignored on every supported version now. They used to
go through `testSparkVersionLE33`, which ran the test on 3.3 and ignored it
everywhere else, so this PR does not change whether any of them execute. It
does make that state permanent, and I would like the CH maintainers' read
before picking a direction, since it is a question about CH coverage rather
than about the version checks this PR removes.
What the 13 cover: nine mergetree bucket-table write cases
(`GlutenClickHouseMergeTreeWriteSuite`, `...OnHDFSSuite`,
`...OnHDFSWithRocksDBMetaSuite`, `...OnS3Suite`, `...OptimizeSuite`,
`...PathBasedWriteSuite`), `base64` in `GlutenClickhouseStringFunctionsSuite`,
the posexplode case from #1767, and the query loop in
`GlutenParquetFilterSuite`.
Two directions: make them pass on 3.5 and drop this helper, or delete the
cases and accept that CH bucket writes lose their coverage. I did not want to
choose either one inside a version-check cleanup.
##########
backends-clickhouse/src/test/scala/org/apache/gluten/execution/GlutenFunctionValidateSuite.scala:
##########
@@ -699,17 +699,10 @@ class GlutenFunctionValidateSuite extends
GlutenClickHouseWholeStageTransformerS
}
test("test common subexpression eliminate") {
+ // TODO: the expected operator counts at the call sites below only held on
Spark 3.3. Re-derive
Review Comment:
The body of this helper sat entirely inside `if (spark33)`, so the eight
call sites below have asserted nothing since 3.3 stopped being built. I kept
the call sites and their expected counts behind the TODO rather than deleting
them, which means `test("test common subexpression eliminate")` still runs and
still reports green while checking nothing about the plan.
Whether those counts still hold on 3.5 needs a ClickHouse run, which I
cannot do here. Either re-derive them and restore the assertion, or drop the
helper and the counts. The same question applies to the copy in
`GlutenClickHouseHiveTableSuite`.
##########
backends-clickhouse/src/test/scala/org/apache/gluten/execution/hive/GlutenClickHouseHiveTableSuite.scala:
##########
@@ -1038,25 +1038,13 @@ class GlutenClickHouseHiveTableSuite
spark.sql(
s"CREATE FUNCTION my_add as " +
s"'org.apache.hadoop.hive.contrib.udf.example.UDFExampleAdd2' USING
JAR '$jarUrl'")
- if (isSparkVersionLE("3.3")) {
- runQueryAndCompare("select MY_ADD(id, id+1) from range(10)")(
- checkGlutenPlan[ProjectExecTransformer])
- } else {
- runQueryAndCompare("select MY_ADD(id, id+1) from range(10)", noFallBack
= false)(_ => {})
- }
+ runQueryAndCompare("select MY_ADD(id, id+1) from range(10)", noFallBack =
false)(_ => {})
}
+ // TODO: the expected operator counts at the call sites below only held on
Spark 3.3. Re-derive
Review Comment:
Same situation as the copy in `GlutenFunctionValidateSuite`: the body was
entirely inside `if (spark33)`, so the CSE tests below run green while
asserting nothing about the plan. Keeping the expected counts behind a TODO
preserves the information, but restoring the assertion needs a ClickHouse run
to re-derive them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
