sunchao opened a new pull request, #59972: URL: https://github.com/apache/spark/pull/59972
### What changes were proposed in this pull request? Allow nested scan-schema pruning through dynamic Map keys and array indices, including `element_at`. For example, `m[key_column].id` can request only `id` from the Map values while retaining `key_column` and the Map keys. Reuse the existing dependency traversal to collect every input used by the key/index expression, including inputs beneath the same collection root and supported array-function predicates. Rebind dynamic `element_at` expressions to the narrowed schema. No new configuration, dependency, or public API is introduced. ### Why are the changes needed? [SPARK-60766](https://issues.apache.org/jira/browse/SPARK-60766) [SPARK-42163](https://issues.apache.org/jira/browse/SPARK-42163) restricted pruning through collection lookups to constant selectors because the index/key could itself reference a field that pruning would discard. Tracking those dependencies separately allows the reader to skip unused value/element fields safely. For `m: map<string, struct<id:int, payload:string>>`, `SELECT m[key_column].id` currently retains `payload` in the read schema. This change retains only `id` in the values, along with the inputs needed for the lookup. The same applies to `items[index_column].id`. ### Does this PR introduce _any_ user-facing change? Eligible Parquet/ORC scans request fewer nested fields. SQL results, null behavior, and ANSI/index errors are intended to remain unchanged. This prunes fields within collection values; it does not read only the matching Map entry or array element. ### How was this patch tested? Added coverage for selector dependencies under null checks and beneath the same collection root, field-ordinal rebinding, null/out-of-range/negative/zero array indices, supported array-function compositions, and renamed root/nested Parquet field IDs. Updated the existing SPARK-42163 regression to assert the narrower read schema. Local targeted validation passed: - 53 Catalyst tests (`SelectedFieldSuite` and `SchemaPruningSuite`). - 26 Parquet tests covering dynamic selectors, the updated SPARK-42163 regression, and renamed root/nested field IDs, across V1/V2 and vectorized/row readers. - A paired control using the exact base production sources failed all 8 updated SPARK-42163 cases at the intended read-schema assertion: the base retains `first,middle,last`, while the candidate requests only `first`. - Changed-file Scalastyle: 6 files, 0 errors, 0 warnings. `git diff --check` passed. These checks freshly compiled the current production/test sources over public Spark 4.1.3 dependencies. An isolated test-helper inheritance shim was needed for the older runtime; no production or actual test source was rewritten. This is targeted source-overlay validation, not a full master build. Full-build validation runs in fork CI. ORC tests were not run locally. No throughput benchmark or measured latency improvement is claimed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
