ravjot28 opened a new pull request, #19872:
URL: https://github.com/apache/hudi/pull/19872

   ### Describe the issue this Pull Request addresses
   
   An Avro payload decoded with a projected schema can lose or misassign values 
when the requested schema differs from the schema that encoded the record. 
Repeated projections, metadata-declaring schemas, nested renames, and 
serialization round trips expose this in merge paths.
   
   This is a prerequisite for the standalone Java work in [discussion 
#19798](https://github.com/apache/hudi/discussions/19798) and the [dev-list 
proposal](https://www.mail-archive.com/dev%40hudi.apache.org/msg06536.html).
   
   ### Summary and Changelog
   
   - Retain the writer schema and resolve projections, aliases, defaults, and 
supported legacy positional renames without shifting unrelated fields.
   - Restrict positional renames to compatible source fields that are not 
already claimed by another reader field or alias. Incompatible drop/add 
replacements receive their reader default.
   - Avoid inferring a legacy writer schema from a prefix read that leaves 
unread record bytes. Later full-schema reads can still recover trailing values.
   - Preserve the writer schema through Java serialization and default Kryo 
serialization. Cache compressed schema strings when smaller, keeping each 
serialized payload independently readable across threads and random-access 
spills.
   - Preserve the original Java serialVersionUID and accept golden legacy 
streams. Keep Spark's registered compact Kryo representation and its existing 
size assertions unchanged.
   - Add regression coverage for the review cases, compressed/uncompressed 
transport, tombstones, and Spark Kryo without the Hudi registrar. Split 
recursive reconciliation into named helpers.
   
   This uses Hudi's Avro rewrite utilities, Apache Avro's compatibility 
checker, and JDK gzip/Base64. No dependency or third-party source is added.
   
   ### Impact
   
   Shared payload decoding affects every engine using these payloads. No new 
table storage layout or table version is introduced.
   
   **Breaking transport change (the `!` in the title):** default non-Spark Kryo 
writers emit a schema-bearing representation with a negative-length 
discriminator. The schema string contains JSON or `gzip:` followed by 
Base64-encoded compressed JSON. Updated readers accept legacy 
non-negative-length payloads, uncompressed schema-bearing payloads, and 
compressed schema-bearing payloads. Pre-change readers cannot read the 
schema-bearing representation; readers from an earlier revision of this PR that 
only understand JSON cannot read its compressed form.
   
   `BaseAvroPayload.useLegacyKryoFormat(kryo)` retains the old representation 
for transports supplying the writer schema separately. The Spark Hudi registrar 
selects that mode. Mixed-version users must upgrade readers first or explicitly 
retain legacy transport; arbitrary mixed-version compatibility is not claimed. 
Legacy bytes remain schema-free: consuming all bytes avoids caching a truncated 
prefix, but does not prove historical field names/types.
   
   Compression reduces repeated schema overhead without introducing 
stream-local IDs. A sparse 128-column size probe changed from 8,016 to 729 
bytes per serialized payload (146 bytes in legacy mode). This is a size 
measurement, not an end-to-end performance claim; compression and schema 
parsing are cached with a 1,024-entry bound.
   
   ### Risk Level
   
   high
   
   Shared merge behavior and the default in-memory Kryo representation change. 
Review regressions were reproduced against the previous PR head before the 
fixes. Local validation in the Hudi Java 11 CI image passed 70 focused cases 
with zero failures, errors, skips or retry events: 26 payload cases, four 
shared serializer cases, 32 existing disk-spill cases, five Spark 
registrar/default-serializer cases, and the three unchanged Spark 
record-serialization/size cases. The dependency reactor builds successfully; 
Checkstyle passes for both changed modules. Unrelated ScalaTest suites were 
excluded from this targeted run. This is not a full repository test run. The 
previous head `a8530e348f` passed [Azure 
CI](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16927);
 that result does not validate this revision.
   
   ### Documentation Update
   
   The transport boundary and migration requirements are documented above. The 
prepared companion change for the `asf-site` branch will document this in 
`website/docs/java-client.md` and be linked before merge.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to