cshuo commented on code in PR #19200:
URL: https://github.com/apache/hudi/pull/19200#discussion_r3532926657
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/cdc/HoodieNativeLogFormatWriter.java:
##########
@@ -237,23 +234,12 @@ private Map<HeaderMetadataType, String>
addRecordPositionsIfRequired(
return header;
}
- Set<Long> positionSet = new HashSet<>(recordPositions.size());
- long previousPosition = Long.MIN_VALUE;
- for (Long position : recordPositions) {
- // RECORD_POSITIONS is encoded as a bitmap and decoded in ascending
order. Native log records
- // are read back in file write order, so only publish positions when
both orders match.
- if (!HoodieRecordLocation.isPositionValid(position) || position <=
previousPosition) {
- Map<HeaderMetadataType, String> updatedHeader = new HashMap<>(header);
-
updatedHeader.remove(HeaderMetadataType.BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS);
- return updatedHeader;
- }
- previousPosition = position;
- positionSet.add(position);
- }
-
Map<HeaderMetadataType, String> updatedHeader = new HashMap<>(header);
- if (positionSet.size() == recordPositions.size()) {
- updatedHeader.put(HeaderMetadataType.RECORD_POSITIONS,
LogReaderUtils.encodePositions(positionSet));
+ Option<String> encodedRecordPositions =
LogReaderUtils.encodePositionsAsLongList(recordPositions);
Review Comment:
The old duplicate guard was needed for the bitmap format because duplicate
positions would be collapsed by the set/bitmap encoding of
Roaring64NavigableMap, causing the decoded position count to diverge from the
record count. That no longer applies to native data/delete blocks with the
ordered position list format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
