yyanyy commented on code in PR #55252:
URL: https://github.com/apache/spark/pull/55252#discussion_r3054403087
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:
##########
@@ -787,14 +800,22 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan]
with PredicateHelper {
val wrappedScan = getWrappedScan(scan, sHolder)
- val scanRelation = DataSourceV2ScanRelation(sHolder.relation,
wrappedScan, output)
-
val projectionOverSchema =
ProjectionOverSchema(output.toStructType, AttributeSet(output))
val projectionFunc = (expr: Expression) => expr transformDown {
case projectionOverSchema(newExpr) => newExpr
}
+ // Remap pushed filter attributes to the pruned output schema. Note: if
a pushed filter
Review Comment:
It actually could happen since in V2 a fully pushed filter can be removed
from the plan and no longer be referenced, triggering column pruning to drop
it. But Wenchen also pointed a similar issue later, that I'll share some more
details in that comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
