Re: now throttling indexing
Using ES 1.4.4 I should add. On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote: I have the same problem. On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote: I set `indices.store.throttle.type: none` in the elasticsearch.yml, and yet this shows up in the logs: now throttling indexing: numMergesInFlight=5, maxNumMerges=4 stop throttling indexing: numMergesInFlight=3, maxNumMerges=4 Did I misunderstand the purpose of this setting? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: now throttling indexing
I have the same problem. On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote: I set `indices.store.throttle.type: none` in the elasticsearch.yml, and yet this shows up in the logs: now throttling indexing: numMergesInFlight=5, maxNumMerges=4 stop throttling indexing: numMergesInFlight=3, maxNumMerges=4 Did I misunderstand the purpose of this setting? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f1e4959b-1de9-45fe-ac9a-00c31268fb30%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: now throttling indexing
That is the right setting to disable store throttling, but even without throttling writes MB/sec for merges, the merges can still fall behind, leading to index throttling. ES does this to protect the health of the index because too many segments will cause all sorts of trouble. What IO system is your index on? If you're on spinning disks you could try setting index.merge.scheduler.max_thread_count to 1 since spinning disks struggle with concurrent merges. See http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html Also, do you leave enough (at least 50%) free RAM to the OS for buffering pages? This can make a difference with spinning disks since the OS has more freedom to do read-ahead on the files being merged... Mike McCandless http://www.elastic.co On Fri, Mar 13, 2015 at 1:01 AM, Alexander Jiteg alexander.ji...@tickstar.com wrote: Using ES 1.4.4 I should add. On Friday, March 13, 2015 at 9:00:26 AM UTC+1, Alexander Jiteg wrote: I have the same problem. On Friday, March 13, 2015 at 2:16:33 AM UTC+1, Eric Jain wrote: I set `indices.store.throttle.type: none` in the elasticsearch.yml, and yet this shows up in the logs: now throttling indexing: numMergesInFlight=5, maxNumMerges=4 stop throttling indexing: numMergesInFlight=3, maxNumMerges=4 Did I misunderstand the purpose of this setting? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/c7498550-47bb-40a5-ae1b-e01173d11efc%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKHUQPgPG9ovisOJ0q_zQaZ-HoNK0pWsqrhC9r-z5gzBhoQ6JQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: now throttling indexing
On Fri, Mar 13, 2015 at 6:09 AM, Michael McCandless m...@elastic.co wrote: That is the right setting to disable store throttling, but even without throttling writes MB/sec for merges, the merges can still fall behind, leading to index throttling. ES does this to protect the health of the index because too many segments will cause all sorts of trouble. What IO system is your index on? If you're on spinning disks you could try setting index.merge.scheduler.max_thread_count to 1 since spinning disks struggle with concurrent merges. See http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html Also, do you leave enough (at least 50%) free RAM to the OS for buffering pages? This can make a difference with spinning disks since the OS has more freedom to do read-ahead on the files being merged... The index is on an SSD (EC2 m3.large). I'll look into what else is going on, had just noticed a few of those log entries, and thought setting `indices.store.throttle.type: none` might be a quick fix :-) -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2B%2BqrQzoQAicwz9Q%2BMprcNVhVqXWKGp_CkKYefKNeWnO_A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
now throttling indexing
I set `indices.store.throttle.type: none` in the elasticsearch.yml, and yet this shows up in the logs: now throttling indexing: numMergesInFlight=5, maxNumMerges=4 stop throttling indexing: numMergesInFlight=3, maxNumMerges=4 Did I misunderstand the purpose of this setting? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3b0f57e1-f78e-4782-ad4d-2dfe42bb5c17%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
