xiangfu0 commented on code in PR #20223:
URL: https://github.com/apache/pinot/pull/20223#discussion_r4538294637
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -2278,6 +2278,12 @@ private void setConsumeEndTime(SegmentZKMetadata
segmentZKMetadata, long now) {
long maxConsumeTimeMillis = _streamConfig.getFlushThresholdTimeMillis();
_consumeEndTime = segmentZKMetadata.getCreationTime() +
maxConsumeTimeMillis;
+ // For pauseless tables, keep the end time from ZK while it is in the
future so that all replicas stop together.
+ // A replica that consumes past the end offset fixed at commit start can
only download the committed segment.
+ if (_consumeEndTime > now &&
PauselessConsumptionUtils.isPauselessEnabled(_tableConfig)) {
Review Comment:
If `PartitionConsumer.run()` starts consuming after this end time (for
example while it waits on `isReadyToConsumeData` for upsert/dedup at startup,
or on the consumer semaphore), the first `endCriteriaReached()` sees `now >=
_consumeEndTime` with `_hasMessagesFetched == false` and adds 1 hour to the end
time. I reproduced this locally: with creationTime = now-29m, a 30m threshold
and a 2-min delay before `consumeLoop()`, the end time became creationTime+90m
(before this PR it was creationTime+39m with no extension). When other replicas
are healthy, the CONSUMING→ONLINE transition still stops this replica. With
replication 1 or after a full restart, though, the segment runs about 1h past
the threshold. Could the empty-segment extension be skipped until the first
fetch, or the shared end time be applied when consumption actually starts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
