voonhous commented on code in PR #19478:
URL: https://github.com/apache/hudi/pull/19478#discussion_r3709745608
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/split/HudiSplitFactory.java:
##########
@@ -127,7 +127,10 @@ private static List<HudiSplit> createSplitsForBaseFile(
}
ImmutableList.Builder<HudiSplit> splits = ImmutableList.builder();
- long targetSplitSizeInBytes = Math.max(targetSplitSize.toBytes(),
baseFile.getPathInfo().getBlockSize());
+ // Slicing is governed solely by the target split size; the block size
reported by
+ // storage is not meaningful on object stores and must not influence
split sizing.
+ long targetSplitSizeInBytes = targetSplitSize.toBytes();
+ checkArgument(targetSplitSizeInBytes > 0, "targetSplitSize must be
positive: %s", targetSplitSize);
Review Comment:
Pulled it into this PR rather than a follow-up, since it hardens the exact
knob this PR repurposes:
- `@MinDataSize("1B")` on `HudiConfig.getTargetSplitSize`
- `validateMinDataSize(TARGET_SPLIT_SIZE, value, DataSize.ofBytes(1))` on
the session property, same shape as `parquet_small_file_threshold`
`0B` now fails at config/session time with `target_split_size must be at
least 1B: 0B`. The constructor `checkArgument` stays as a backstop for
programmatic construction. Covered by
`TestHudiConfig.testTargetSplitSizeValidation` and
`TestHudiSessionProperties.testSessionPropertyTargetSplitSizeRejectsZero`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
