hudi-agent commented on code in PR #19083:
URL: https://github.com/apache/hudi/pull/19083#discussion_r3757150552
##########
hudi-io/src/test/java/org/apache/hudi/io/hfile/TestHFileWriter.java:
##########
@@ -200,84 +231,124 @@ void testUniqueKeyLocation() throws IOException {
);
reader.next();
}
+
+ // Each data block's previous-block-offset header (8 bytes at offset 16)
must chain back to
+ // the prior block, and -1 for the first, so an HBase reader's
seekBefore can step back across
+ // blocks. The writer once set it to the block's own offset; this guards
that regression.
+ byte[] file = Files.readAllBytes(Paths.get(testFile));
+ List<IndexEntry> dataIndex = parseIndexBlock(
+ file, (int) trailer.getLoadOnOpenDataOffset(),
trailer.getDataIndexCount());
+ assertTrue(dataIndex.size() > 1, "test setup: expected multiple data
blocks");
+ for (int i = 0; i < dataIndex.size(); i++) {
+ long prevOffset = readLongBE(file, (int) dataIndex.get(i).offset + 16);
+ long expected = i == 0 ? -1L : dataIndex.get(i - 1).offset;
+ assertEquals(expected, prevOffset, "data block " + i + "
previous-block offset");
+ }
}
}
+ /**
+ * Format lock for the native writer's on-disk bytes. Single-block inputs
are checked against the
+ * HBase writer (proving the writers agree on the encoding): short keys
(single-byte index
+ * keyLength) and the 116-char boundary (keyLength 128, where Protobuf and
Hadoop VInt diverge) are
+ * also pinned to a golden; 500- and 2000-char first keys (3-byte keyLength)
are checked against
+ * HBase without a golden. The golden cases also validate the trailer and
file info bytes. A single
+ * block is deliberate for cross-writer checks: across blocks HBase stores
shortened index separator
+ * keys, so only a single full-key entry is byte-comparable. A multi-block
native file then covers
+ * the meta block and the rest of the section byte layout and round-trips.
+ */
@Test
- void testLongKeys() throws IOException {
- // Test that HFile blocks with long keys (>= 126 chars) can be written and
read correctly.
- // This verifies the fix for the varint encoding mismatch in the root
index block.
- HFileContext context = new HFileContext.Builder().blockSize(100).build();
- String testFile = TEST_FILE;
- int numRecords = 10;
- // Generate keys longer than 126 characters to trigger multi-byte Hadoop
VarInt encoding
- // in the root index block. The varint encodes (key_content_length + 2),
so content >= 126
- // produces a value >= 128 which requires 2+ bytes in Hadoop VarInt format.
- char[] chars = new char[200];
- Arrays.fill(chars, 'a');
- String longPrefix = new String(chars);
- try (DataOutputStream outputStream =
- new DataOutputStream(Files.newOutputStream(Paths.get(testFile)));
- HFileWriter writer = new HFileWriterImpl(context, outputStream)) {
- for (int i = 0; i < numRecords; i++) {
- String key = longPrefix + String.format("%04d", i);
- writer.append(key, String.format("value%04d", i).getBytes());
- }
- }
+ void writerBlockBytesAreStableFormatLock() throws Exception {
+ byte[][] vals = {bytes("v0"), bytes("v1"), bytes("v2")};
+ assertSingleBlockBytesMatchGoldenAndHBase(
+ new String[] {"key1", "key2", "key3"},
+ new byte[][] {bytes("value1"), bytes("value2"), bytes("value3")},
Review Comment:
🤖 nit: `vals` is a bit terse here — could you rename it to `shortValues` or
`threeShortValues` to distinguish it at a glance from the `values` arrays in
the first call (which use `"value1"` etc.)?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
