klsince commented on code in PR #15296:
URL: https://github.com/apache/pinot/pull/15296#discussion_r1999305231
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1723,6 +1725,15 @@ public RealtimeSegmentDataManager(SegmentZKMetadata
segmentZKMetadata, TableConf
}
}
+ // Consumption while building the segment is not allowed for the following
tables:
+ // 1. Partial Upserts
Review Comment:
For full upsert, other than that all records are appended to the table, the
records with same PK use comparison columns to decide the record that's valid
to queries and mark all others as invalid with the validDocIds bitmaps, so the
order of record ingestion does not matter for data correctness.
For partial upsert, partial record is merged with prev record, so ingestion
order matters across replicas for data consistency. For dedup, there is no
validDocIds bitmaps to invalidate ingested records, so slow replicas need to
wait for committed segments to be downloaded&replaced to get dedup metadata up
to date, before starting new ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
