OK, I would suggest setting index.merge.scheduler.max_thread_count to 1 for spinning disks.
Maybe try also disabling merge throttling and see if that has an effect? 6 MB/sec seems slow... Mike McCandless http://blog.mikemccandless.com On Mon, Aug 25, 2014 at 8:57 PM, Chris Decker <ch...@chris-decker.com> wrote: > Mike, > > Thanks for the response. > > I'm running ES 1.2.1. It appears the issue that you reported / corrected > was included with ES 1.2.0. > > *Any other ideas / suggestions? *Were the settings that I posted sane? > > > Thanks!, > Chris > > On Monday, August 25, 2014 1:52:46 PM UTC-4, Michael McCandless wrote: > >> Which version of ES are you using? Versions before 1.2 have a bug that >> caused merge throttling to throttle far more than requested such that you >> couldn't get any faster than ~8 MB / sec. See https://github.com/ >> elasticsearch/elasticsearch/issues/6018 >> >> Tiered merge policy is best. >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> >> On Mon, Aug 25, 2014 at 1:08 PM, Chris Decker <ch...@chris-decker.com> >> wrote: >> >>> All, >>> >>> I’m looking for advice on how to reduce the number of segments for my >>> indices because in my use case (log analysis), quick searches are more >>> important than real-time access to data. I've turned many of the "knobs" >>> available within ES, and read many blog postings, ES documentation, etc., >>> but still feel like there is room for important. >>> >>> Specific questions I have: >>> 1. How can I increase the current merge rate? According to Elastic HQ, >>> my merge rate is 6 MB/s (according to Elastic HQ). I know I don't have >>> SSDs, but with 15k drives it seems like I should be able to get better >>> rates. I tried increasing indices.store.throttle.max_bytes_per_sec >>> from the default of 20mb to 40mb in my templates, but I didn't see a >>> noticeable change in disk IOps or the merge rate the next day. Did I do >>> something incorrectly? I'm going to experiment with setting it overall >>> with index.store.throttle.max_bytes_per_sec and removing it from my >>> templates. >>> 2. Should I move away from the default merge policy, or stick with the >>> default ("tiered")? >>> >>> Any advice you have is much appreciated; additional details on my >>> situation are below. >>> >>> ---- >>> >>> - I generate 2 indices per day - “high” and “low”. I usually end up >>> with ~ 450 segments for my ‘high’ index (see attached), and another ~ 200 >>> segments for my ‘low’ index, which I then optimize once I roll-over to the >>> next day’s indices. >>> - 4 ES servers (soon to be 8). >>> — Each server has: >>> 12 Xeon cores running at 2.3 GHz >>> 15k drives >>> 128 GB of RAM >>> 68 GB used for OS / file system machine >>> 60 GB used by 2 JVMs >>> - Index ~ 750 GB per day; 1.5 TB if you include the replicas >>> - Relevant configs: >>> TEMPLATE: >>> "index.refresh_interval" : "60s", >>> "index.number_of_replicas" : "1", >>> "index.number_of_shards" : "4", >>> "index.merge.policy.max_merged_segment" : "50g", >>> "index.merge.policy.segments_per_tier" : "5", >>> "index.merge.policy.max_merge_at_once" : “5”, >>> "indices.store.throttle.max_bytes_per_sec" : "40mb". >>> >>> ELASTICSEARCH.YML: >>> indices.memory.index_buffer_size: 30% >>> >>> >>> >>> Thanks in advance!, >>> Chris >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/46ecc658-502f-46c7-b2b9-db9fd0e9f58f%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/46ecc658-502f-46c7-b2b9-db9fd0e9f58f%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRdE%2B1%3DijK2nycH-sqXvjqVEQ%3DZvGo65YfozHpDTSOZCVg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
