You interpreted my suggestion correctly.
Elasticsearch can indeed leverage information that is sent to the
InfoStream: look up the class called LoggerInfoStream, which forwards all
the InfoStream logging to Log4J.
On Wed, Sep 22, 2021 at 1:40 PM Chris Hegarty
<christopher.hega...@elastic.co.invalid> wrote:
> Hi Adrien,
>
> Great suggestion. If I understand correctly, then what you are suggesting
> is something along the lines of ( subject to exact message details, which
> we can trash out in a PR ):
>
> *diff --git
> a/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java
> b/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java*
> *index 4500d5cf7ce..76f7ea2a9f2 100644*
> *---
> a/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java*
> *+++
> b/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java*
> @@ -691,13 +691,23 @@ public class ConcurrentMergeScheduler extends
> MergeScheduler {
> public void run() {
> try {
> if (verbose()) {
> - message(" merge thread: start");
> + message(String.format(Locale.ROOT, "merge thread %s start",
> this.getName()));
> }
>
>
> doMerge(mergeSource, merge);
>
>
> if (verbose()) {
> - message(" merge thread: done");
> + message(
> + String.format(
> + Locale.ROOT,
> + "merge thread %s done estSize=%.1f MB (written=%.1f MB)
> runTime=%.1fs (stopped=%.1fs, paused=%.1fs) rate=%s",
> + this.getName(),
> + bytesToMB(merge.estimatedMergeBytes),
> + bytesToMB(rateLimiter.getTotalBytesWritten()),
> + nsToSec(System.nanoTime() - merge.mergeStartNS),
> + nsToSec(rateLimiter.getTotalStoppedNS()),
> + nsToSec(rateLimiter.getTotalPausedNS()),
> + rateToString(rateLimiter.getMBPerSec())));
> }
> runOnMergeFinished(mergeSource);
> } catch (Throwable exc) {
>
>
> Then ES can leverage such from the infoStream, right? ( thus avoiding the
> need for ES extract the inaccessible information directly itself, while
> also being more generally useful in Lucene logs ).
>
> Or have I misinterpreted your comment?
>
> -Chris.
>
>
> On 22 Sep 2021, at 10:12, Adrien Grand <jpou...@gmail.com> wrote:
>
> Hi Chris,
>
> I looked into this and Elasticsearch seems to only need access to the rate
> limiter for logging purposes, without adding any information that Lucene
> doesn't have.
> Maybe another option would consist of moving the logging to Lucene? Having
> information in the IndexWriter's InfoStream about rate limiting for each
> completed merge sounds like something that would generally be useful.
>
>
>
--
Adrien
