praveenc7 commented on code in PR #15585:
URL: https://github.com/apache/pinot/pull/15585#discussion_r2055097104
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java:
##########
@@ -214,8 +217,19 @@ public Response downloadSegment(
segmentFile =
org.apache.pinot.common.utils.FileUtils.concatAndValidateFile(tableDir,
segmentName + "-" + UUID.randomUUID(),
"Invalid segment name: %s", segmentName);
-
+ long deepStoreDownloadStartTimeMs = System.currentTimeMillis();
pinotFS.copyToLocalFile(remoteSegmentFileURI, segmentFile);
+ long deepStoreDownloadDurationMs = System.currentTimeMillis() -
deepStoreDownloadStartTimeMs;
+ _controllerMetrics.addTimedTableValue(tableName,
ControllerTimer.SEGMENT_DEEP_STORE_DOWNLOAD_TIME_MS,
Review Comment:
We wanted to have table-level metrics to get finer understanding, you are
right global level can be inferred from merging the table-level. But since the
global would be just 1 extra metric, we taught of including it here to have a
raw global level metric
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -434,9 +438,22 @@ protected void
replaceSegmentIfCrcMismatch(SegmentDataManager segmentDataManager
@Override
public void downloadAndLoadSegment(SegmentZKMetadata zkMetadata,
IndexLoadingConfig indexLoadingConfig)
throws Exception {
+ long downloadStartTimeMs = System.currentTimeMillis();
String segmentName = zkMetadata.getSegmentName();
_logger.info("Downloading and loading segment: {}", segmentName);
+ // Download the segment
File indexDir = downloadSegment(zkMetadata);
+ long totalDownloadDurationMs = System.currentTimeMillis() -
downloadStartTimeMs;
+ _serverMetrics.addTimedValue(_rawTableName,
ServerTimer.SEGMENT_TOTAL_DOWNLOAD_TIME_MS,
+ totalDownloadDurationMs, TimeUnit.MILLISECONDS);
+ _serverMetrics.addTimedValue(ServerTimer.SEGMENT_TOTAL_DOWNLOAD_TIME_MS,
totalDownloadDurationMs,
+ TimeUnit.MILLISECONDS);
+ // Load the segment
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig,
_segmentOperationsThrottler));
+ long loadDurationMs = System.currentTimeMillis() - downloadStartTimeMs -
totalDownloadDurationMs;
Review Comment:
NA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
