dragonknightk opened a new pull request, #1025:
URL: https://github.com/apache/hudi-rs/pull/1025

   ## Description
   
   Part of #908
   
   `decode_record_index_entry` called `from_avro_datum` for every record, and 
that path re-resolves the full `HoodieMetadataRecord` schema, including every 
index payload type, on each call. Record-index entries have a fixed layout, so 
this change uses direct decode when the container writer schema passes the 
layout check, and keeps generic decode as the fallback. The fallback to generic 
decode has two triggers: the layout check fails, or direct decode fails. The 
layout check is computed once per HFile. On a release build, decoding one 
record-index entry drops from about 8.8 us to about 0.45 us (four sample shapes 
rotated for 100000 iterations).
   
   The layout check compares field names, order and types to the vendored 
`HoodieMetadata.avsc`. A null union branch is one byte regardless of the other 
branch. Record-index records leave the other payloads null, so a difference in 
the column-stats wrapper union still passes. An extra or reordered top-level 
field, or a change to `HoodieRecordIndexInfo` itself, fails the layout check, 
because it can change the bytes.
   
   Swapping `fileIdHighBits` and `fileIdLowBits` keeps the same field types, so 
direct decode would still succeed but return a file ID with its two halves 
swapped. The layout check rejects that schema.
   
   Column-stats encode and decode are left alone, as issue 908 step 1 asked.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
     - [ ] Details are described below
   
   - Direct decode and generic decode agree field by field, including null and 
default `HoodieRecordIndexInfo` fields and a Spark-compacted record-index HFile 
fixture.
   - Empty bytes and a null `recordIndexMetadata` stay deletes.
   - The layout check accepts a column-stats-only change and rejects mismatches.
   - `decode_record_index_entry` matches generic decode for each of those 
schemas, and when direct decode fails after a passing layout check.
   - Truncated prefixes and fixed corrupt inputs do not panic, and 
`decode_record_index_entry` matches generic decode.
   - `apply_hfile_records` matches generic decode on a schema that fails the 
layout check.
   
   ## AI Disclosure
   
   Human-led contribution. Grok Bot, working for dragonknightk, assisted with 
this change, including the code, tests, and description. All submitted content 
was reviewed by a human before submission.
   
   (๑•̀ㅂ•́)و✧
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to