Really we need the infoStream output, to see what IW is doing, to take so long merging.
Likely only one merge thread is running (CMS tries to detect if your IO system "spins" and if so, uses 1 merge thread) ... maybe try configuring this to something higher since your RAID array can probably handle it? It's good that disabling auto IO throttling didn't change things ... that's what I expected (since forced merges are not throttled by default). Maybe capture all thread stacks and post back here? Mike McCandless http://blog.mikemccandless.com On Thu, Jun 16, 2016 at 4:04 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 6/16/2016 2:35 AM, Michael McCandless wrote: > > > > Hmm, merging can't read at 800 MB/sec and only write at 20 MB/sec for > > very long ... unless there is a huge percentage of deletes. Also, by > > default CMS doesn't throttle forced merges (see > > CMS.get/setForceMergeMBPerSec). Maybe capture > > IndexWriter.setInfoStream output? > > I can see the problem myself. I have a RAID10 array with six SATA > disks. When I click the Optimize button for a core that's several > gigabytes, iotop shows me reads happening at about 100MB/s for several > seconds, then writes clocking no more than 25 MB/s, and usually a lot > less. The last several gigabytes that were written were happening at > less than 5 MB/s. This is VERY slow, and does affect my nightly > indexing processes. > > Asking the shell to copy a 5GB file revealed sustained write rates of > over 500MB/s, so the hardware can definitely go faster. > > I patched in an option for solrconfig.xml where I could force it to call > disableAutoIOThrottle(). I included logging in my patch to make > absolutely sure that the new code was used. This option made no > difference in the write speed. I also enabled infoStream, but either I > configured it wrong or I do not know where to look for the messages. I > was modifying and compiling branch_5_5. > > This is the patch that I applied: > > http://apaste.info/wKG > > I did see the expected log entries in solr.log when I restarted with the > patch and the new option in solrconfig.xml. > > What else can I look at? > > Thanks, > Shawn > >
