shounakmk219 commented on code in PR #19170:
URL: https://github.com/apache/pinot/pull/19170#discussion_r3793972799
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1448,28 +1448,34 @@ public void ensureAllPartitionsConsuming(TableConfig
tableConfig, List<StreamCon
Preconditions.checkState(!_isStopping, "Segment manager is stopping");
String realtimeTableName = tableConfig.getTableName();
+
+ // Fetch all stream offsets BEFORE acquiring the ideal-state update lock.
On tables with many partitions these
+ // stream round-trips can take minutes; doing them here (against a
snapshot of the ideal state) keeps the
+ // per-table ideal-state lock hold-time proportional to the in-memory
ideal-state mutation, not to the offset
+ // I/O. The updater lambda below performs no stream I/O, so it is also
cheap to re-run on ZK CAS retries.
+ IdealState snapshotIdealState =
HelixHelper.getTableIdealState(_helixManager, realtimeTableName);
Review Comment:
only the offset pre-fetch inputs come from the snapshot, every read that
drives a repair decision or a written offset is from the fresh IS under the
lock. We fetch the latestSegmentZKMetadataMap again as part of
ensureAllPartitionsConsuming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
