This is an automated email from the ASF dual-hosted git repository.
baibaichen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 27aa68207a [GLUTEN-12569][CORE] Add enhanced DSv2 filter test suites
for Spark 4.2 (#13164)
27aa68207a is described below
commit 27aa68207aff4d734cec2605044f0e25801309df
Author: Akshay Tayal <[email protected]>
AuthorDate: Thu Oct 15 19:32:29 2026 -0700
[GLUTEN-12569][CORE] Add enhanced DSv2 filter test suites for Spark 4.2
(#13164)
* [GLUTEN-12569][CORE] Add enhanced runtime partition filter tests for
Spark 4.2
Wrap Spark 4.2's DataSourceV2EnhancedRuntimePartitionFilterSuite with the
Gluten test trait and enable it for Velox. This adds 15 upstream tests for
the iterative runtime partition-filter path used by BatchScanExec.
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5
(cherry picked from commit ed127752e36e74820f061da7478cfee0c105c571)
* [GLUTEN-12569][CORE] Add enhanced partition and delete filter tests for
Spark 4.2
Wrap Spark 4.2's DataSourceV2EnhancedPartitionFilterSuite and
DataSourceV2EnhancedDeleteFilterSuite with the Gluten test trait and enable
them for Velox, completing coverage of the three new enhanced DSv2 filter
suites.
This adds 37 upstream tests covering first/second-pass partition filter
pushdown, untranslatable and nested partition predicates, post-scan filter
retention, and delete filter handling.
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5
(cherry picked from commit 6a8449800685c72cdffefe0593a38828e3f8497a)
* [GLUTEN-12569][VL] Disable 2 CI-failing enhanced DSv2 partition-filter
UTs for Spark 4.2
GlutenDataSourceV2EnhancedPartitionFilterSuite 'case 9 ...' (+ nested
variant):
the test asserts a post-scan vanilla FilterExec remains, but Gluten
replaces it
(columnar transform) / the InMemoryEnhancedPartitionFilterBatchScan falls
back,
so the plan-shape assertion fails. CI-confirmed (apache #13164 group3).
Disabled
via VeloxTestSettings.exclude; the other enhanced suites (Delete,
RuntimePartition)
pass.
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 413738d2-8cf1-4c70-9331-6c5e4314f9fa
* [GLUTEN-12569][VL] Rewrite the 2 case-9 enhanced partition-filter UTs as
Gluten tests
The vanilla "case 9" tests in DataSourceV2EnhancedPartitionFilterSuite
assert
that a Spark FilterExec remains above the scan for the partition filter the
source returns in the first pass. With Gluten, the in-memory DSv2 scan falls
back but that post-scan filter is offloaded to a native
FilterExecTransformer, so the vanilla assertion fails even though the plan
is
correct.
Keep the vanilla tests excluded and add Gluten variants that keep the
answer,
second-pass pushdown and scanned-partition checks, and accept either a Spark
FilterExec or a Gluten FilterExecTransformerBase as the post-scan filter.
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 3a1d1e30-b32c-45b1-9f0e-d7aa4654eab9
* [GLUTEN-12569][VL] Check the returned equality in the Gluten case-9
post-scan filter
Address review feedback: the Gluten case-9 tests only checked that a
post-scan filter exists. Require the query's filter condition, taken from
the analyzed plan, to be one of the conjuncts of the post-scan filter, so
a filter that keeps only IsNotNull no longer passes. Works for both
FilterExec and FilterExecTransformerBase.
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 3a1d1e30-b32c-45b1-9f0e-d7aa4654eab9
---------
Co-authored-by: Akshay Tayal <[email protected]>
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5
Copilot-Session: 413738d2-8cf1-4c70-9331-6c5e4314f9fa
Copilot-Session: 3a1d1e30-b32c-45b1-9f0e-d7aa4654eab9
---
.../gluten/utils/velox/VeloxTestSettings.scala | 9 ++
...utenDataSourceV2EnhancedDeleteFilterSuite.scala | 23 +++++
...nDataSourceV2EnhancedPartitionFilterSuite.scala | 111 +++++++++++++++++++++
...urceV2EnhancedRuntimePartitionFilterSuite.scala | 23 +++++
4 files changed, 166 insertions(+)
diff --git
a/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
b/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
index 155babfed1..2023bf3236 100644
---
a/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
+++
b/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
@@ -57,6 +57,15 @@ class VeloxTestSettings extends BackendTestSettings {
enableSuite[GlutenBloomFilterAggregateQuerySuiteCGOff]
enableSuite[GlutenDataSourceV2DataFrameSessionCatalogSuite]
enableSuite[GlutenDataSourceV2DataFrameSuite]
+ enableSuite[GlutenDataSourceV2EnhancedDeleteFilterSuite]
+ enableSuite[GlutenDataSourceV2EnhancedPartitionFilterSuite]
+ .exclude(
+ "case 9: partition filter pushed but returned in first pass is not
re-pushed second pass"
+ ) // Rewrite to also accept a native post-scan filter.
+ .exclude(
+ "nested identity partition: case 9 partition filter pushed but returned
in first pass is not re-pushed second pass"
+ ) // Rewrite to also accept a native post-scan filter.
+ enableSuite[GlutenDataSourceV2EnhancedRuntimePartitionFilterSuite]
enableSuite[GlutenDataSourceV2FunctionSuite]
enableSuite[GlutenDataSourceV2SQLSessionCatalogSuite]
enableSuite[GlutenDataSourceV2SQLSuiteV1Filter]
diff --git
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedDeleteFilterSuite.scala
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedDeleteFilterSuite.scala
new file mode 100644
index 0000000000..cc470f09ce
--- /dev/null
+++
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedDeleteFilterSuite.scala
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.GlutenSQLTestsTrait
+
+class GlutenDataSourceV2EnhancedDeleteFilterSuite
+ extends DataSourceV2EnhancedDeleteFilterSuite
+ with GlutenSQLTestsTrait {}
diff --git
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedPartitionFilterSuite.scala
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedPartitionFilterSuite.scala
new file mode 100644
index 0000000000..452f509aaf
--- /dev/null
+++
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedPartitionFilterSuite.scala
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connector
+
+import org.apache.gluten.execution.FilterExecTransformerBase
+
+import org.apache.spark.sql.{DataFrame, GlutenSQLTestsTrait, Row}
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.connector.catalog.{BufferedRows,
InMemoryEnhancedPartitionFilterTable}
+import org.apache.spark.sql.execution.ExplainUtils.stripAQEPlan
+import org.apache.spark.sql.execution.FilterExec
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+
+class GlutenDataSourceV2EnhancedPartitionFilterSuite
+ extends DataSourceV2EnhancedPartitionFilterSuite
+ with GlutenSQLTestsTrait {
+
+ // The vanilla "case 9" tests (excluded in VeloxTestSettings) require a
Spark FilterExec for the
+ // partition filter that the source returns in the first pass, but Gluten
can offload that
+ // post-scan filter to a native FilterExecTransformer. These variants keep
the answer and
+ // pushdown checks and accept either filter.
+ testGluten(
+ "case 9: partition filter pushed but returned in first pass is not
re-pushed second pass") {
+ withTable(partFilterTableName) {
+ sql(
+ s"CREATE TABLE $partFilterTableName (part_col string, data string)
USING $v2Source " +
+ "PARTITIONED BY (part_col) " +
+ "TBLPROPERTIES('return-accepted-partition-predicates' = 'true')")
+ sql(s"INSERT INTO $partFilterTableName VALUES ('a', 'x'), ('b', 'y'),
('c', 'z')")
+
+ val df = sql(s"SELECT * FROM $partFilterTableName WHERE part_col = 'b'")
+ checkAnswer(df, Seq(Row("b", "y")))
+ assertNoSecondPassPartitionPredicates(df)
+ assertScannedPartitionKeys(df, Set("b"))
+ assertPostScanFilter(df)
+ }
+ }
+
+ testGluten(
+ "nested identity partition: case 9 partition filter pushed but returned in
first pass " +
+ "is not re-pushed second pass") {
+ withTable(partFilterTableName) {
+ sql(
+ s"CREATE TABLE $partFilterTableName " +
+ s"(s struct<tz: string, x: int>, data string) USING $v2Source " +
+ "PARTITIONED BY (s.tz) " +
+ "TBLPROPERTIES('return-accepted-partition-predicates' = 'true')")
+ sql(
+ s"INSERT INTO $partFilterTableName VALUES " +
+ "(named_struct('tz', 'LA', 'x', 1), 'a'), " +
+ "(named_struct('tz', 'NY', 'x', 2), 'b')")
+
+ val df = sql(s"SELECT * FROM $partFilterTableName WHERE s.tz = 'LA'")
+ checkAnswer(df, Seq(Row(Row("LA", 1), "a")))
+ assertNoSecondPassPartitionPredicates(df)
+ assertScannedPartitionKeys(df, Set("LA"))
+ assertPostScanFilter(df)
+ }
+ }
+
+ private def batchScan(df: DataFrame): BatchScanExec = {
+ stripAQEPlan(df.queryExecution.executedPlan)
+ .collectFirst { case b: BatchScanExec => b }
+ .getOrElse(fail("Expected BatchScanExec in plan"))
+ }
+
+ private def assertNoSecondPassPartitionPredicates(df: DataFrame): Unit = {
+ val predicates = batchScan(df).batch match {
+ case s:
InMemoryEnhancedPartitionFilterTable#InMemoryEnhancedPartitionFilterBatchScan =>
+ s.getPushedPartitionPredicates
+ case _ => Seq.empty
+ }
+ assert(predicates.isEmpty, s"Expected no pushed partition predicates, got
$predicates")
+ }
+
+ private def assertScannedPartitionKeys(df: DataFrame, expectedKeys:
Set[String]): Unit = {
+ val partitions = batchScan(df).batch.planInputPartitions()
+ assert(partitions.length === expectedKeys.size)
+ assert(partitions.map(_.asInstanceOf[BufferedRows].keyString()).toSet ===
expectedKeys)
+ }
+
+ private def assertPostScanFilter(df: DataFrame): Unit = {
+ val expected = df.queryExecution.analyzed
+ .collectFirst { case f: Filter => f.condition }
+ .getOrElse(fail("Expected a Filter in the analyzed plan"))
+ val hasReturnedPredicate =
stripAQEPlan(df.queryExecution.executedPlan).exists {
+ case f: FilterExec =>
+
splitConjunctivePredicates(f.condition).exists(_.semanticEquals(expected))
+ case f: FilterExecTransformerBase =>
+ splitConjunctivePredicates(f.cond).exists(_.semanticEquals(expected))
+ case _ => false
+ }
+ assert(
+ hasReturnedPredicate,
+ "Partition filter returned in first pass should remain in the post-scan
filter")
+ }
+}
diff --git
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedRuntimePartitionFilterSuite.scala
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedRuntimePartitionFilterSuite.scala
new file mode 100644
index 0000000000..c3b36209e9
--- /dev/null
+++
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenDataSourceV2EnhancedRuntimePartitionFilterSuite.scala
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.GlutenSQLTestsTrait
+
+class GlutenDataSourceV2EnhancedRuntimePartitionFilterSuite
+ extends DataSourceV2EnhancedRuntimePartitionFilterSuite
+ with GlutenSQLTestsTrait {}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
