On 6/16/2016 2:35 AM, Michael McCandless wrote: > > Hmm, merging can't read at 800 MB/sec and only write at 20 MB/sec for > very long ... unless there is a huge percentage of deletes. Also, by > default CMS doesn't throttle forced merges (see > CMS.get/setForceMergeMBPerSec). Maybe capture > IndexWriter.setInfoStream output?
I can see the problem myself. I have a RAID10 array with six SATA disks. When I click the Optimize button for a core that's several gigabytes, iotop shows me reads happening at about 100MB/s for several seconds, then writes clocking no more than 25 MB/s, and usually a lot less. The last several gigabytes that were written were happening at less than 5 MB/s. This is VERY slow, and does affect my nightly indexing processes. Asking the shell to copy a 5GB file revealed sustained write rates of over 500MB/s, so the hardware can definitely go faster. I patched in an option for solrconfig.xml where I could force it to call disableAutoIOThrottle(). I included logging in my patch to make absolutely sure that the new code was used. This option made no difference in the write speed. I also enabled infoStream, but either I configured it wrong or I do not know where to look for the messages. I was modifying and compiling branch_5_5. This is the patch that I applied: http://apaste.info/wKG I did see the expected log entries in solr.log when I restarted with the patch and the new option in solrconfig.xml. What else can I look at? Thanks, Shawn
