danny0405 commented on code in PR #19973:
URL: https://github.com/apache/hudi/pull/19973#discussion_r4025501390
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java:
##########
@@ -1033,6 +1037,29 @@ public void testCacheCleanOfRecordIndexBackend() throws
Exception {
.checkWrittenData(EXPECTED1);
}
+ @Test
+ public void testIndexWriteFunctionWithSmallBuffer() throws Exception {
+ conf.set(FlinkOptions.INDEX_TYPE,
HoodieIndex.IndexType.GLOBAL_RECORD_LEVEL_INDEX.name());
+
conf.setString(HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key(),
"true");
+ conf.setString(HoodieMetadataConfig.STREAMING_WRITE_ENABLED.key(), "true");
+ conf.set(FlinkOptions.INDEX_RLI_WRITE_BUFFER_SIZE, 1L);
+
+ StreamWriteFunctionWrapper<RowData> pipeline =
+ (StreamWriteFunctionWrapper<RowData>)
TestData.getWritePipeline(tempFile.getAbsolutePath(), conf);
+ pipeline.openFunction();
+ try {
+ // The record cannot fit even after flushing and resetting the 1 MB
index buffer.
+ RowData indexRow = GenericRowData.of(IndexRowUtils.RLI_TYPE,
+ StringData.fromString("k".repeat(2 * 1024 * 1024)),
+ StringData.fromString("par1"), StringData.fromString("file1"));
+ HoodieException exception = assertThrows(HoodieException.class,
+ () -> pipeline.getIndexWriteFunction().processElement(indexRow,
null, null));
+ assertEquals("Index write buffer is too small to hold a single record.",
exception.getMessage());
Review Comment:
Non-blocking: could we also cover the successful retry after a
capacity-triggered flush? This test exercises a record that cannot fit in an
empty buffer, while the existing checkpoint tests flush explicitly. A companion
test with individually fitting records whose combined size exceeds the buffer
could verify that the earlier records are flushed and the record that triggered
the flush is retained and eventually written to the index. That would protect
the other outcome of the retry check, including against accidentally throwing
on the first failed write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
