WangGuangxin opened a new pull request, #58303:
URL: https://github.com/apache/spark/pull/58303
### What changes were proposed in this pull request?
This PR enables Parquet predicate pushdown for shredded Variant fields in
the DSv2 scan path.
The existing shredded Variant predicate pushdown support can translate
filters on `variant_get`
expressions into filters on shredded Variant struct fields, so Parquet can
use them for row-group
skipping. However, DSv2 regular filter pushdown happens before Variant
extraction pushdown, so these
rewritten filters are not available during the normal `pushDownFilters`
phase.
This PR adds a narrow internal pushdown path for this case:
- Adds an internal `SupportsPushDownVariantPredicateFilters` hook for scan
builders.
- Rewrites remaining filters after Variant extraction pushdown and
translates the rewritten filters
with nested predicate pushdown enabled.
- Lets `ParquetScanBuilder` accept these Variant predicate filters only
when both
`spark.sql.parquet.filterPushdown` and
`spark.sql.variant.shreddedPredicatePushdown.enabled` are enabled.
- Carries accepted filters through `ParquetScan` into
`ParquetPartitionReaderFactory`.
- Combines regular Parquet filters with accepted Variant predicate filters
only when the read schema
contains shredded Variant metadata, and then reuses the existing
`ParquetFilters` shredded Variant
logic to build Parquet row-group predicates.
- Keeps the original Spark filters above the scan, so this is only a scan
pruning optimization and
does not change query results.
### Why are the changes needed?
Without this change, DSv2 Parquet scans can push Variant extraction into
the scan, but predicates
that become pushdown-able only after that rewrite are not passed to
Parquet. As a result, queries
filtering on shredded Variant fields may still read row groups that
Parquet could otherwise skip.
This is especially useful for Variant columns stored with shredding, where
common paths are materialized
as typed Parquet fields. Pushing these predicates to Parquet lets Spark
avoid unnecessary IO while
preserving the existing post-scan filter semantics.
### Does this PR introduce _any_ user-facing change?
No.
This only improves Parquet row-group skipping for eligible DSv2 scans when
existing pushdown-related
configuration is enabled. Query results and visible SQL semantics are
unchanged.
### How was this patch tested?
Updated `VariantShreddingFilterPushdownSuite` to cover DSv2 shredded
Variant predicate pushdown,
including row-group skipping assertions for vectorized reads and
correctness coverage across DSv1/DSv2
and vectorized/non-vectorized readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
