szehon-ho commented on code in PR #58524:
URL: https://github.com/apache/spark/pull/58524#discussion_r3938157815
##########
sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala:
##########
@@ -2680,6 +2680,70 @@ abstract class DynamicPartitionPruningV2FilterSuite
super.initState()
spark.conf.set("spark.sql.catalog.testcat",
classOf[InMemoryTableWithV2FilterCatalog].getName)
}
+
+ import testImplicits._
+
+ private def collectFactScan(df: DataFrame): BatchScanExec = {
+ val scans = collectWithSubqueries(df.queryExecution.executedPlan) {
+ case b: BatchScanExec if b.runtimeFilters.nonEmpty => b
+ }
+ assert(scans.size == 1, s"expected exactly 1 scan with runtime filters,
got:\n$scans")
+ scans.head
+ }
+
+ test("SPARK-59250: DPP prunes all DSv2 partitions when the runtime IN filter
is empty",
Review Comment:
Could we also cover the `SupportsRuntimeFiltering` (V1 `Filter`) path? This
PR updates `InMemoryBaseTable.InMemoryBatchScan` to handle `AlwaysFalse`, but
these tests run only with `InMemoryTableWithV2FilterCatalog`, so they exercise
`InMemoryV2FilterBatchScan` and leave the V1 compatibility path untested. It
may be possible to share the tests with `DynamicPartitionPruningV2Suite` so
both implementations run them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
