deepthi912 commented on code in PR #19394:
URL: https://github.com/apache/pinot/pull/19394#discussion_r3911426735
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java:
##########
@@ -158,6 +158,16 @@ public static void preprocess(File indexDir,
IndexLoadingConfig indexLoadingConf
Preconditions.checkArgument(indexDir.isDirectory(), "Index directory: %s
does not exist or is not a directory",
indexDir);
+ // Respect skipSegmentPreprocess uniformly across every caller — including
cold-download load paths
Review Comment:
You're right, and I'll switch to that pattern. My earlier push-back was
framed around minion-built / batch cold-download where the fresh segment still
needs its indexes built — but the pauseless-realtime case (which is really what
this fix targets) is the opposite: the committer already ran preprocess before
uploading, so non-committer replicas downloading the tar get a fully
preprocessed segment and `needPreprocess()` would correctly return `false` and
save the whole preprocess call.
The check is per-load but cheap; the work it gates is expensive. Paying the
check universally is the safer contract and matches how `BaseTableDataManager`
warm-load / reload paths already do it.
I'll update the fix to route through `needPreprocess(segmentDirectory,
indexLoadingConfig)` at the call site so the cold-download callers use the same
contract as the warm paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
