zhangyue19921010 commented on a change in pull request #10676: URL: https://github.com/apache/druid/pull/10676#discussion_r544091714
########## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java ########## @@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments) } } + /** + * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a + * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on + * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available + * for querying. + * + * @param toolbox {@link TaskToolbox} object with for assisting with task work. + * @param segmentsToWaitFor {@link List} of segments to wait for availability. + * @param waitTimeout Millis to wait before giving up + * @return True if all segments became available, otherwise False. + */ + protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout) + { + if (segmentsToWaitFor.isEmpty()) { + log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?"); + return false; + } + log.info("Waiting for segments to be loaded by the cluster..."); + + SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory() Review comment: nit: Each time we call the `waitForSegmentAvailability` function will communicate with coordinator no matter what the value of `waitTimeout` is and bring extra pressure to the Coordiantor, because there may hundreds of thousands of batch tasks per day. Maybe we can do a **double check** here, like when `waitTimeout<=0` then skip all the `waitForSegmentAvailability ` if possible , just in case that call `waitForSegmentAvailability` function without checking `waitTimeout`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org