Mike, OK, thanks. I had looked at setting index.merge.scheduler.max_thread_count in the past but thought the default was already 1; I see now that it was increased. Thanks for the heads-up!
I realized after sending my last reply that I’m actually getting about a 12MB/sec per server (6MB/sec per node); I failed to sum the rate for the 2 nodes on each server. With throttling disabled altogether, I did notice it jump up to about 14MB/sec, but it wasn’t substantial. Any other recommendations? I’m still seeing a significant number of segments (~ 400) created for my ‘high’ daily index. FWIW, I have another 4 servers on order which should help the situation, but I want to make sure I’m taking full advantage of my resources. Thanks, Chris From: Michael McCandless <m...@elasticsearch.com> Reply: elasticsearch@googlegroups.com <elasticsearch@googlegroups.com>> Date: August 26, 2014 at 4:27:31 PM To: elasticsearch@googlegroups.com <elasticsearch@googlegroups.com>> Subject: Re: Reduce Number of Segments OK, I would suggest setting index.merge.scheduler.max_thread_count to 1 for spinning disks. Maybe try also disabling merge throttling and see if that has an effect? 6 MB/sec seems slow... Mike McCandless http://blog.mikemccandless.com On Mon, Aug 25, 2014 at 8:57 PM, Chris Decker <ch...@chris-decker.com> wrote: Mike, Thanks for the response. I'm running ES 1.2.1. It appears the issue that you reported / corrected was included with ES 1.2.0. Any other ideas / suggestions? Were the settings that I posted sane? Thanks!, Chris On Monday, August 25, 2014 1:52:46 PM UTC-4, Michael McCandless wrote: Which version of ES are you using? Versions before 1.2 have a bug that caused merge throttling to throttle far more than requested such that you couldn't get any faster than ~8 MB / sec. See https://github.com/elasticsearch/elasticsearch/issues/6018 Tiered merge policy is best. Mike McCandless http://blog.mikemccandless.com On Mon, Aug 25, 2014 at 1:08 PM, Chris Decker <ch...@chris-decker.com> wrote: All, I’m looking for advice on how to reduce the number of segments for my indices because in my use case (log analysis), quick searches are more important than real-time access to data. I've turned many of the "knobs" available within ES, and read many blog postings, ES documentation, etc., but still feel like there is room for important. Specific questions I have: 1. How can I increase the current merge rate? According to Elastic HQ, my merge rate is 6 MB/s (according to Elastic HQ). I know I don't have SSDs, but with 15k drives it seems like I should be able to get better rates. I tried increasing indices.store.throttle.max_bytes_per_sec from the default of 20mb to 40mb in my templates, but I didn't see a noticeable change in disk IOps or the merge rate the next day. Did I do something incorrectly? I'm going to experiment with setting it overall with index.store.throttle.max_bytes_per_sec and removing it from my templates. 2. Should I move away from the default merge policy, or stick with the default ("tiered")? Any advice you have is much appreciated; additional details on my situation are below. ---- - I generate 2 indices per day - “high” and “low”. I usually end up with ~ 450 segments for my ‘high’ index (see attached), and another ~ 200 segments for my ‘low’ index, which I then optimize once I roll-over to the next day’s indices. - 4 ES servers (soon to be 8). — Each server has: 12 Xeon cores running at 2.3 GHz 15k drives 128 GB of RAM 68 GB used for OS / file system machine 60 GB used by 2 JVMs - Index ~ 750 GB per day; 1.5 TB if you include the replicas - Relevant configs: TEMPLATE: "index.refresh_interval" : "60s", "index.number_of_replicas" : "1", "index.number_of_shards" : "4", "index.merge.policy.max_merged_segment" : "50g", "index.merge.policy.segments_per_tier" : "5", "index.merge.policy.max_merge_at_once" : “5”, "indices.store.throttle.max_bytes_per_sec" : "40mb". ELASTICSEARCH.YML: indices.memory.index_buffer_size: 30% Thanks in advance!, Chris -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/002cb4cc-fa2e-43c3-b2d3-29580742c91a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46ecc658-502f-46c7-b2b9-db9fd0e9f58f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/QoW-LSz3QUI/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRdE%2B1%3DijK2nycH-sqXvjqVEQ%3DZvGo65YfozHpDTSOZCVg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53ff579d.6b8b4567.e6%40april.sos.its.psu.edu. For more options, visit https://groups.google.com/d/optout.
