hudi-agent commented on code in PR #20319: URL: https://github.com/apache/hudi/pull/20319#discussion_r4278124479
########## hudi-io/hfile_format.md: ########## @@ -443,6 +536,24 @@ Here are the meaning of each field: - `compression_codec`: Compression codec: 0 = LZO, 1 = GZ, 2 = NONE - `encryption_key`: Encryption key (not used by Hudi) +The four offsets (`file_info_offset`, `load_on_open_data_offset`, `first_data_block_offset`, +`last_data_block_offset`) are `uint64`, so the trailer can address any position in a file larger than 4 GiB; only the Review Comment: 🤖 I think "only the counts are `uint32`" doesn't quite match the proto above, since `entry_count` is a `uint64` count and `num_data_index_levels`/`compression_codec` are `uint32` but aren't counts; could you name the `uint32` fields directly instead? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## hudi-io/hfile_format.md: ########## @@ -321,15 +328,101 @@ many. With one level, the Root Data Index Block in the "Load-on-open" section po blocks, and that is all Hudi's writer ever produces: it writes a single root level and sets `num_data_index_levels` to `1`. -With more than one level, the root index entries point at index blocks rather than data blocks. Those -intermediate levels use the `IDXINTE2` (`INTERMEDIATE_INDEX`) magic and the bottom level uses `IDXLEAF2` -(`LEAF_INDEX`); both carry the same block index entry layout described above. HBase produces these once a -file has more data blocks than fit in a single root index. +With more than one level, the root index entries point at index blocks rather than data blocks. HBase +produces these once a file has more data blocks than fit in a single root index. The bottom level uses the +`IDXLEAF2` (`LEAF_INDEX`) magic and any level between the root and the leaves uses `IDXINTE2` +(`INTERMEDIATE_INDEX`). HBase writes each leaf index block inline in the "Scanned block" section, right after +the data blocks it covers; intermediate index blocks go after the meta blocks in the "Non-scanned block" +section; the root stays in the "Load-on-open" section. A two-level file looks like this: + +``` ++----------+-----------------------+ +| "Scanned | Data Block 0 | +| block" +-----------------------+ +| section | Data Block 1 | +| +-----------------------+ +| | Data Block 2 | +| +-----------------------+ +| | Leaf Index Block 0 | --> Data Blocks 0, 1, 2 +| +-----------------------+ +| | Data Block 3 | +| +-----------------------+ +| | Data Block 4 | +| +-----------------------+ +| | Leaf Index Block 1 | --> Data Blocks 3, 4 ++----------+-----------------------+ +| "Non- | Meta Block | +| scanned +-----------------------+ +| block" | ... | +| section | | ++----------+-----------------------+ +| "Load- | Root Data Index Block | --> Leaf Index Blocks 0, 1 Review Comment: 🤖 Would it make sense to note that in a multi-level file HBase appends mid-key metadata after the root data index entries (mid leaf block offset, its on-disk size, and the mid-key entry index, 16 bytes total), so readers don't expect the root block's data to end right after the last entry? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
