Hi. I am studying Lucene performance and in particular how it benefits from
faster I/O such as SSD and NVMe.
I am using nightlybench for indexing wiki (1K docs) with similar parameters as
used in nightlyBench. (Hardware: Intel Xeon, 2.5GHz, 20 processor ,40 with
hyperthreading, 64G Memory) and study indexing speed on HDD, SSD and NVMe.
While I do see benefit when switching from HDD to SSD, there is not much
noticeable benefit moving to NVMe.
I get best performance (200GB/hour) with 20 indexing threads, increasing number
of threads to 40 hurts performance. Similarly increasing maxConcurrentMerges
above 3-5 doesn't seem to give me any benefit. I am wondering what the
bottleneck is, or anyone has insight on set of options (number of threads,
merge options, flush options, read buffer?) to take advantage of a very fast
I/O system. I see NVMe bandwidth going as high as 800MB/s but it is only fast
spikes and CPU utilization is about 50% on average, though some cores have
consistently higher utilization while others have spiky behavior.
You thoughts and inside is greatly appreciated. Thanks.
Anahita Shayesteh
