voonhous opened a new pull request, #20160: URL: https://github.com/apache/hudi/pull/20160
### Describe the issue this Pull Request addresses branch-1.2.x does not have #20041 (issue #20040). On Spark 4.1, where `spark.sql.variant.pushVariantIntoScan` is on by default, a query that pushes several paths of one variant, most of them absent from a row, fails with `NegativeArraySizeException`: - on a MOR read that merges a log file, with either record merger; - on a bootstrap read that joins the skeleton and the source file. The record context builds its row writers from the engine schema, which types the column as `VariantType`, while the rows carry the projection struct. The partial-update merge reads the base record through the same types. Reads with the rule off are correct. ### Summary and Changelog Hand-port of master `76be6c65b2c1`. - `SparkRecordMergingUtils`, `BaseSparkInternalRecordContext`, `BaseSparkInternalRowReaderContext`: applied as on master. - `SparkFileFormatInternalRowReaderContext`: resolved by hand. This branch projects log records for every merger, so `setSchemaHandler` installs the row shape whenever the query carries a projection, without master's payload-merger exclusion. - `TestVariantPushVariantIntoScan`: a wide projection and a partial-update `MERGE INTO` on the top-level sweep, and a bootstrap test (COW and MOR, both arms). <details> <summary>Local runs, Spark 4.1.1</summary> | State | Result | |:--|:--| | branch-1.2.x, tests only | top-level sweep fails on the two MOR rule-on legs (`NegativeArraySizeException`), bootstrap test fails (`NegativeArraySizeException`) | | port without the `SparkRecordMergingUtils` hunk | `MERGE INTO` check fails on the two MOR rule-on legs (`ClassCastException`: `VariantVal` to `InternalRow`) | | port | 3 tests, 16 sweep legs, all pass | Not ported: master's time-travel and incremental legs, which sit in a suite this branch does not have. </details> ### Impact Spark 4.1+ reads with the rule on. No change with the rule off or on Spark 3.x and 4.0, where no projection struct reaches the reader. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
