J-HowHuang commented on code in PR #18174:
URL: https://github.com/apache/pinot/pull/18174#discussion_r3113798227
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -926,17 +926,18 @@ protected void
setZkOperationTimeIfAvailable(ImmutableSegment segment, @Nullable
}
}
- private void reloadSegments(List<SegmentDataManager> segmentDataManagers,
IndexLoadingConfig indexLoadingConfig,
+ private void reloadSegmentDataManagersInParallel(List<SegmentDataManager>
segmentDataManagers,
boolean forceDownload, String reloadJobId)
throws Exception {
+ IndexLoadingConfig indexLoadingConfigTemplate = fetchIndexLoadingConfig();
List<String> failedSegments = new ArrayList<>();
AtomicReference<Throwable> sampleException = new AtomicReference<>();
CompletableFuture.allOf(segmentDataManagers.stream().map(segmentDataManager ->
CompletableFuture.runAsync(() -> {
String segmentName = segmentDataManager.getSegmentName();
try {
_segmentReloadSemaphore.acquire(segmentName, _logger);
try {
- reloadSegment(segmentDataManager, indexLoadingConfig, forceDownload);
+ reloadSegment(segmentDataManager, indexLoadingConfigTemplate.copy(),
forceDownload);
Review Comment:
> Since offline reload mutates IndexLoadingConfig (tier, setTableDataDir,
etc.), so the same instance can’t be used concurrently without coordination.
I don't get this. Once you make a copy of index loading config per tier, you
can then set tier and table data dir on the copy for each tier, and there's no
need to mutate them within each reload task, no? Synchronizing the reload
operation here is a significant performance degradation, which is definitely
not preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
