nsivabalan commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2244003475
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>>
computeRevivedAndDeletedKeys(Set<St
* @return
*/
@VisibleForTesting
- public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord>
recordIndexRecords, int parallelism) {
- return recordIndexRecords.mapToPair(
- (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>)
t -> Pair.of(t.getKey(), t))
- .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord,
HoodieRecord>) (record1, record2) -> {
- boolean isRecord1Deleted = record1.getData() instanceof
EmptyHoodieRecordPayload;
- boolean isRecord2Deleted = record2.getData() instanceof
EmptyHoodieRecordPayload;
- if (isRecord1Deleted && !isRecord2Deleted) {
- return record2;
- } else if (!isRecord1Deleted && isRecord2Deleted) {
- return record1;
- } else if (isRecord1Deleted && isRecord2Deleted) {
- // let's delete just 1 of them
- return record1;
- } else {
- throw new HoodieIOException("Two HoodieRecord updates to RLI is
seen for same record key " + record2.getRecordKey() + ", record 1 : "
- + record1.getData().toString() + ", record 2 : " +
record2.getData().toString());
- }
- }, parallelism).values();
+ public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord>
recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+ HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+ if (isPartitionedRLI) {
+ recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+ String recordPartitionPath = r.getData() instanceof
EmptyHoodieRecordPayloadWithPartition
Review Comment:
yes. one high level goal we had while designing the partitioned RLI is:
the lookup we do in MDT rli file groups should be proportional to the data
table hudi partition.
for eg,
if p1 is 1GB and p2 is 100GB.
the number of file groups we lookup in MDT RLI should be relative.
if we take the approach you are suggesting, then even for p1, we might be
polling 100s of RLI file groups.
then we can't really show benefits from RLI in all cases where different
data table partitions have diff sizes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
