sunchao opened a new pull request, #59997: URL: https://github.com/apache/spark/pull/59997
### What changes were proposed in this pull request? Use the original Parquet file schema to distinguish two-level and three-level LIST encodings after clipping. For an unmatched field ID, the row reader discards any physical child read solely to tell whether its parent is null. Clipping and row materialization use the same scan-specific field-ID setting. ### Why are the changes needed? Pruning a struct element to a single field can make a two-level LIST look like a three-level LIST of primitives. A constant subscript such as `legacy[1].renamed` can then fail to read. When Spark requests a missing ID, an unrelated physical field with the same name can also supply a value that should be NULL. These are existing reader bugs identified in the [review of SPARK-60766](https://github.com/apache/spark/pull/59972#discussion_r4433280742). This change can be backported independently. ### Does this PR introduce _any_ user-facing change? Yes. Affected Parquet projections read the correct array element, respect field IDs, and preserve the distinction between a null parent and a present parent with missing fields. ### How was this patch tested? Five focused tests passed against Spark master's 5.0.0-SNAPSHOT CI build with the standalone patch. The same tests on the unmodified parent failed for the two-level LIST in V1 and V2 and passed the other three controls. A further 207 Parquet field-ID, schema, inference, and I/O tests passed. Changed-file ScalaStyle and `git diff --check` passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
