Hi Adrien, Great suggestion. If I understand correctly, then what you are suggesting is something along the lines of ( subject to exact message details, which we can trash out in a PR ):
diff --git a/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java b/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java index 4500d5cf7ce..76f7ea2a9f2 100644 --- a/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java +++ b/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java @@ -691,13 +691,23 @@ public class ConcurrentMergeScheduler extends MergeScheduler { public void run() { try { if (verbose()) { - message(" merge thread: start"); + message(String.format(Locale.ROOT, "merge thread %s start", this.getName())); } doMerge(mergeSource, merge); if (verbose()) { - message(" merge thread: done"); + message( + String.format( + Locale.ROOT, + "merge thread %s done estSize=%.1f MB (written=%.1f MB) runTime=%.1fs (stopped=%.1fs, paused=%.1fs) rate=%s", + this.getName(), + bytesToMB(merge.estimatedMergeBytes), + bytesToMB(rateLimiter.getTotalBytesWritten()), + nsToSec(System.nanoTime() - merge.mergeStartNS), + nsToSec(rateLimiter.getTotalStoppedNS()), + nsToSec(rateLimiter.getTotalPausedNS()), + rateToString(rateLimiter.getMBPerSec()))); } runOnMergeFinished(mergeSource); } catch (Throwable exc) { Then ES can leverage such from the infoStream, right? ( thus avoiding the need for ES extract the inaccessible information directly itself, while also being more generally useful in Lucene logs ). Or have I misinterpreted your comment? -Chris. > On 22 Sep 2021, at 10:12, Adrien Grand <jpou...@gmail.com> wrote: > > Hi Chris, > > I looked into this and Elasticsearch seems to only need access to the rate > limiter for logging purposes, without adding any information that Lucene > doesn't have. > Maybe another option would consist of moving the logging to Lucene? Having > information in the IndexWriter's InfoStream about rate limiting for each > completed merge sounds like something that would generally be useful. >