Adrien Grand created LUCENE-9535:
------------------------------------
Summary: Investigate recent indexing slowdown for wikimedium
documents
Key: LUCENE-9535
URL: https://issues.apache.org/jira/browse/LUCENE-9535
Project: Lucene - Core
Issue Type: Task
Reporter: Adrien Grand
Nightly benchmarks report a ~10% slowdown for 1kB documents as of September
9th: [http://people.apache.org/~mikemccand/lucenebench/indexing.html].
On that day, we added stored fields in DWPT accounting (LUCENE-9511), so I
first thought this could be due to smaller flushed segments and more merging,
but I still wonder whether there's something else. The benchmark runs with 8GB
of heap, 2GB of RAM buffer and 36 indexing threads. So it's about 2GB/36 = 57MB
of RAM buffer per thread in the worst-case scenario that all DWPTs get full at
the same time. Stored fields account for about 0.7MB of memory, or 1% of the
indexing buffer size. How can a 1% reduction of buffering capacity explain a
10% indexing slowdown? I looked into this further by running indexing
benchmarks locally with 8 indexing threads and 128MB of indexing buffer memory,
which would make this issue even more apparent if the smaller RAM buffer was
the cause, but I'm not seeing a regression and actually I'm seeing similar
number of flushes when I disabled memory accounting for stored fields.
I ran indexing under a profiler to see whether something else could cause this
slowdown, e.g. slow implementations of ramBytesUsed on stored fields writers,
but nothing surprising showed up and the profile looked just like I would have
expected.
Another question I have is why the 4kB benchmark is not affected at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
