cloud-fan commented on code in PR #58145:
URL: https://github.com/apache/spark/pull/58145#discussion_r3859461636
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -201,8 +201,12 @@ class DataSourceV2Strategy(session: SparkSession) extends
Strategy with Predicat
val batchExec = BatchScanExec(relation.output, relation.scan,
runtimeFilters,
relation.ordering, relation.relation.table,
relation.keyGroupedPartitioning)
+ // Advisory filters are kept in the logical Filter for the optimizer
only, and Spark never
+ // evaluates them. See SupportsPushDownCatalystFilters.advisoryFilters.
+ val notEvaluatedFilterSet = ExpressionSet(
+ fullyPushedRuntimeFilters ++ relation.advisoryFilters)
DataSourceV2Strategy.withProjectAndFilter(
- project, postScanFilters.diff(fullyPushedRuntimeFilters),
+ project, postScanFilters.filterNot(notEvaluatedFilterSet.contains),
Review Comment:
**Blocking:**
Advisory filters must be excluded for every supported `Scan` subtype. This
subtraction only runs in the generic batch arm: the earlier V1 and local-scan
arms forward `filters` unchanged, and `withProjectAndFilter` wraps them in
`FilterExec`. Please retain the matched `DataSourceV2ScanRelation` in those
arms and apply the same `ExpressionSet` subtraction there, with focused V1 and
local-scan coverage.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableWithTableSample.scala:
##########
@@ -367,3 +371,68 @@ class InMemoryTableWithLegacyTableSample(
seed: Long): Boolean = true
}
}
+
+/**
+ * Sample table plus Catalyst advisory filters. V1 `SupportsPushDownFilters`
cannot
+ * mix with `SupportsPushDownCatalystFilters` (`pushedFilters` return types
clash),
+ * so this wraps the sample builder. Advisory SQL is property
`advisory-filter`.
Review Comment:
**Nit:**
This sentence is missing the words that connect the advisory SQL to the
property.
```suggestion
* so this wraps the sample builder. The advisory SQL is specified by the
`advisory-filter` property.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
