jihoonson commented on a change in pull request #10371: URL: https://github.com/apache/druid/pull/10371#discussion_r488984636
########## File path: server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java ########## @@ -238,25 +272,102 @@ private CoordinatorStats makeStats(int numCompactionTasks, CompactionSegmentIter { final CoordinatorStats stats = new CoordinatorStats(); stats.addToGlobalStat(COMPACTION_TASK_COUNT, numCompactionTasks); - totalSizesOfSegmentsAwaitingCompactionPerDataSource = iterator.totalRemainingSegmentsSizeBytes(); - totalSizesOfSegmentsAwaitingCompactionPerDataSource.object2LongEntrySet().fastForEach( - entry -> { - final String dataSource = entry.getKey(); - final long totalSizeOfSegmentsAwaitingCompaction = entry.getLongValue(); - stats.addToDataSourceStat( - TOTAL_SIZE_OF_SEGMENTS_AWAITING_COMPACTION, - dataSource, - totalSizeOfSegmentsAwaitingCompaction - ); - } - ); + + // Make sure that the iterator iterate through all the remaining segments so that we can get accurate and correct + // statistics (remaining, skipped, processed, etc.). The reason we have to do this explicitly here is because + // earlier (when we are iterating to submit compaction tasks) we may have ran out of task slot and were not able + // to iterate to the first segment that needs compaction for some datasource. + iterator.flushAllSegments(); Review comment: Yeah it could be expensive, but I think it's fine since the compaction period is supposed to be long enough (30 min by default). It should be documented though that compaction period should be long enough to iterate all datasources. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org