yihua commented on code in PR #20319: URL: https://github.com/apache/hudi/pull/20319#discussion_r4286739762
########## hudi-io/hfile_format.md: ########## @@ -443,6 +536,24 @@ Here are the meaning of each field: - `compression_codec`: Compression codec: 0 = LZO, 1 = GZ, 2 = NONE - `encryption_key`: Encryption key (not used by Hudi) +The four offsets (`file_info_offset`, `load_on_open_data_offset`, `first_data_block_offset`, +`last_data_block_offset`) are `uint64`, so the trailer can address any position in a file larger than 4 GiB; only the Review Comment: Fixed, the sentence now names the `uint64` and `uint32` fields. ########## hudi-io/hfile_format.md: ########## @@ -321,15 +328,101 @@ many. With one level, the Root Data Index Block in the "Load-on-open" section po blocks, and that is all Hudi's writer ever produces: it writes a single root level and sets `num_data_index_levels` to `1`. -With more than one level, the root index entries point at index blocks rather than data blocks. Those -intermediate levels use the `IDXINTE2` (`INTERMEDIATE_INDEX`) magic and the bottom level uses `IDXLEAF2` -(`LEAF_INDEX`); both carry the same block index entry layout described above. HBase produces these once a -file has more data blocks than fit in a single root index. +With more than one level, the root index entries point at index blocks rather than data blocks. HBase +produces these once a file has more data blocks than fit in a single root index. The bottom level uses the +`IDXLEAF2` (`LEAF_INDEX`) magic and any level between the root and the leaves uses `IDXINTE2` +(`INTERMEDIATE_INDEX`). HBase writes each leaf index block inline in the "Scanned block" section, right after +the data blocks it covers; intermediate index blocks go after the meta blocks in the "Non-scanned block" +section; the root stays in the "Load-on-open" section. A two-level file looks like this: + +``` ++----------+-----------------------+ +| "Scanned | Data Block 0 | +| block" +-----------------------+ +| section | Data Block 1 | +| +-----------------------+ +| | Data Block 2 | +| +-----------------------+ +| | Leaf Index Block 0 | --> Data Blocks 0, 1, 2 +| +-----------------------+ +| | Data Block 3 | +| +-----------------------+ +| | Data Block 4 | +| +-----------------------+ +| | Leaf Index Block 1 | --> Data Blocks 3, 4 ++----------+-----------------------+ +| "Non- | Meta Block | +| scanned +-----------------------+ +| block" | ... | +| section | | ++----------+-----------------------+ +| "Load- | Root Data Index Block | --> Leaf Index Blocks 0, 1 Review Comment: Added a paragraph in the multi-level subsection, checked against `HFileBlockIndex#getMidKeyMetadata`: offset and on-disk size of the leaf block holding the mid-key, then the entry index within that block. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
