[
https://issues.apache.org/jira/browse/SPARK-60944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated SPARK-60944:
------------------------------
Description:
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
enforces them.
Add an internal table capability exposing fullyPushedFilterAttributes() for
static filter removal. For scans reporting inferred filters,
DataSourceV2Strategy can omit eligible static filter conjuncts that still match
recorded pushdown predicates and whose accessed fields are all covered by that
declaration, while preserving logical filters for statistics. Sources must
fully enforce query filters over declared columns. Keep Boolean projections
executable, including aliases created by MergeSubplans, and retain static
filters for scans without inference. Support top-level columns and exact nested
struct paths, preserving sibling filters and resolving fields against the
pruned scan schema. Keep the existing runtime scan declaration separate and
unchanged, including its top-level restriction.
Tracking the originally inferred expressions is fragile: BooleanSimplification
can factor them together with residual predicates; MergeSubplans can put branch
predicates into Boolean aliases and rebuild scans; and projection,
normalization, and schema pruning can change references. Recording the original
pushdown predicates and checking declared fields avoids tracking expressions
through those rewrites. Reshaped conjuncts, filters introduced after pushdown,
mixed-column expressions, and inferred predicates inside merge aliases remain
executable. The existing statistics-adjustment code is left for a separate
follow-up.
Original PR and discussion: https://github.com/apache/spark/pull/58145
Implementation: https://github.com/apache/spark/pull/60147
was:
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
enforces them.
Add an internal table capability exposing fullyPushedFilterAttributes() for
static filter removal. For scans reporting inferred filters,
DataSourceV2Strategy can omit eligible static filter conjuncts whose accessed
fields are all covered by that declaration, while preserving logical filters
for statistics. Sources must fully enforce query filters over declared columns.
Keep Boolean projections executable, including aliases created by
MergeSubplans, and retain static filters for scans without inference. Support
top-level columns and exact nested struct paths, preserving sibling filters and
resolving fields against the pruned scan schema. Keep the existing runtime scan
declaration separate and unchanged, including its top-level restriction.
Tracking the originally inferred expressions is fragile: BooleanSimplification
can factor them together with residual predicates; MergeSubplans can put branch
predicates into Boolean aliases and rebuild scans; and projection,
normalization, and schema pruning can change references. Attribute-based
checking avoids expression-provenance bookkeeping. Mixed-column expressions and
inferred predicates inside merge aliases can still be evaluated.
Original PR and discussion: https://github.com/apache/spark/pull/58145
Implementation: https://github.com/apache/spark/pull/60147
> Skip DSv2 filters on fully pushed attributes
> --------------------------------------------
>
> Key: SPARK-60944
> URL: https://issues.apache.org/jira/browse/SPARK-60944
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> DSv2 sources can report inferred Catalyst filters for statistics estimation
> after SPARK-58892, including a mode that keeps them on logical Filter nodes.
> These predicates can remain in executable filters even when the source
> already enforces them.
> Add an internal table capability exposing fullyPushedFilterAttributes() for
> static filter removal. For scans reporting inferred filters,
> DataSourceV2Strategy can omit eligible static filter conjuncts that still
> match recorded pushdown predicates and whose accessed fields are all covered
> by that declaration, while preserving logical filters for statistics. Sources
> must fully enforce query filters over declared columns. Keep Boolean
> projections executable, including aliases created by MergeSubplans, and
> retain static filters for scans without inference. Support top-level columns
> and exact nested struct paths, preserving sibling filters and resolving
> fields against the pruned scan schema. Keep the existing runtime scan
> declaration separate and unchanged, including its top-level restriction.
> Tracking the originally inferred expressions is fragile:
> BooleanSimplification can factor them together with residual predicates;
> MergeSubplans can put branch predicates into Boolean aliases and rebuild
> scans; and projection, normalization, and schema pruning can change
> references. Recording the original pushdown predicates and checking declared
> fields avoids tracking expressions through those rewrites. Reshaped
> conjuncts, filters introduced after pushdown, mixed-column expressions, and
> inferred predicates inside merge aliases remain executable. The existing
> statistics-adjustment code is left for a separate follow-up.
> Original PR and discussion: https://github.com/apache/spark/pull/58145
> Implementation: https://github.com/apache/spark/pull/60147
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
