peterenescu opened a new pull request, #13525: URL: https://github.com/apache/gluten/pull/13525
## What changes are proposed in this pull request? #13109 added ScanMapKeyPruning, which declares the constant map keys a query looks up as required subfields of a native Parquet scan, so the Velox reader skips the other map entries. This PR extends the rule to ORC and DWRF scans. Nothing else had to change. FileSourceScanExecTransformer and BatchScanExecTransformer already carry the declaration for any format. SubstraitToVeloxPlan already turns it into HiveColumnHandle required subfields without looking at the format. On the Velox side, makeScanSpec turns those subfields into a filter on the map keys, and the DWRF reader, which also reads ORC, applies that filter. For a regular map, the key reader drops entries whose key fails the filter, and only the matching values are read. For a DWRF flat map, the reader skips the streams of keys that fail the filter. The rule's only gate was a Parquet format check. It now accepts ORC and DWRF as well and is renamed from isPrunableParquetScan to isPrunableScan. The scaladoc and the scanMapKeyPruningEnabled config doc now say Parquet, ORC or DWRF. HiveTableScanExecTransformer does not support the declaration for any format yet, Parquet included, so this PR leaves it alone. ## How was this patch tested? - ORC cases in ScanMapKeyPruningSuite that mirror the Parquet ones: a keyed filter with aggregates, keyed values projected above a filter, string keys, long keys, a map nested in a struct, and two whole-map uses that must not prune. Each case checks the declared subfields and compares the result with vanilla Spark. - A new test in the same suite attaches a declaration straight to an ORC scan and compares what the scan returns with vanilla Spark running map_filter to the declared keys. It covers string and integer keys, null and empty maps, and maps that hold none of the declared keys. - Spark cannot write DWRF, so a new C++ test, Substrait2VeloxPlanConversionTest.requiredSubfieldsPruneDwrfMapKeys, writes DWRF files with regular and flat-map encoding. It checks that the scan returns every entry without a declaration and exactly the declared entries with one. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
