szehon-ho commented on code in PR #58145:
URL: https://github.com/apache/spark/pull/58145#discussion_r3859785021
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -201,8 +201,12 @@ class DataSourceV2Strategy(session: SparkSession) extends
Strategy with Predicat
val batchExec = BatchScanExec(relation.output, relation.scan,
runtimeFilters,
relation.ordering, relation.relation.table,
relation.keyGroupedPartitioning)
+ // Advisory filters are kept in the logical Filter for the optimizer
only, and Spark never
+ // evaluates them. See SupportsPushDownCatalystFilters.advisoryFilters.
+ val notEvaluatedFilterSet = ExpressionSet(
+ fullyPushedRuntimeFilters ++ relation.advisoryFilters)
DataSourceV2Strategy.withProjectAndFilter(
- project, postScanFilters.diff(fullyPushedRuntimeFilters),
+ project, postScanFilters.filterNot(notEvaluatedFilterSet.contains),
Review Comment:
Thanks, fixed in aa8397a5fde. `DataSourceV2Strategy` now retains the matched
`DataSourceV2ScanRelation` in the V1 and local-scan arms and uses the same
`ExpressionSet`-based subtraction for all three scan paths. I also added
focused V1 and `LocalScan` regression tests; the 10 advisory-filter strategy
tests pass.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala:
##########
@@ -367,3 +371,68 @@ class InMemoryTableWithLegacyTableSample(
seed: Long): Boolean = true
}
}
+
+/**
+ * Sample table plus Catalyst advisory filters. V1 `SupportsPushDownFilters`
cannot
+ * mix with `SupportsPushDownCatalystFilters` (`pushedFilters` return types
clash),
+ * so this wraps the sample builder. Advisory SQL is property
`advisory-filter`.
Review Comment:
Thanks, fixed the wording as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
