On 10/10/2018 11:52 AM, Michael Sokolov wrote:
If maxMergeCount was 2, you could get into a situation with three large merges I think; the largest would be paused, but the others could still take > 10 mins to complete. Are you sure that your observation is at odds with what the document says the scheduler is doing?
I haven't done extremely comprehensive checking, and it has been a number of years now. When I was looking, what appeared to be happening was three merges scheduled. The smallest one I would expect to complete in seconds, or certainly within a few minutes. The largest one was probably at the merge policy's 5GB max segment size, and a merge of that size would definitely take longer than ten minutes. I no longer have access to those indexes, so I can't investigate directly.
There are still new reports on solr-user of database connections failing while importing millions of rows, even recently. I have NOT heard about anyone applying my fix (set maxMergeCount to 6) and still seeing failures, but I suppose that might have happened.
It is the recent reports of the problem that has prompted me to investigate deeper and start this thread. I believe that the merge scheduler SHOULD handle smaller merges first, just like the javadocs indicate, but I have seen evidence (at least in the past) that it's not actually doing so. My look at the code today seems to indicate that it is sorting large merges first, but somebody who's intimately familiar with that code could decipher it a lot faster than I can.
Thanks, Shawn --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
