Hi Elliott, 1500 segments per shard is certainly way too much, and it is not normal that optimize doesn't manage to reduce the number of segments. - Is there anything suspicious in the logs? - Have you customized the merge policy or scheduler?[1] - Does the issue still reproduce if you restart your cluster?
[1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <ebradsh...@gmail.com>wrote: > Any other thoughts on this? Would 1500 segments per shard be > significantly impacting performance? Have you guys noticed this behavior > elsewhere? > > Thanks. > > > On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: >> >> Adrian, >> >> I ran the following command: >> >> curl -XPUT http://localhost:9200/_settings -d >> '{"indices.store.throttle.max_bytes_per_sec" : "10gb"}' >> >> and received a { "acknowledged" : "true" } response. The logs showed >> "cluster state updated". >> >> I did have to close my index prior to changing the setting and reopen >> afterward. >> >> >> I've since began another optimize, but again it doesn't look like much is >> happening. The optimize isn't returning and the total CPU usage on every >> node is holding at about 2% of a single core. I would copy a hot_threads >> stack trace, but I'm unfortunately on a closed network and this isn't >> possible. I can tell you that refreshes of hot_threads show vary little >> happening. The occasional [merge] thread (always in a >> LinkedTransferQueue.awaitMatch() >> state) or [optimize] (doing nothing on a waitForMerge() call) thread shows >> up, but it's always consuming 0-1% CPU. It sure feels like something isn't >> right. Any thoughts? >> >> >> >> >> On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <adrien.grand@elasticsearch. >> com> wrote: >> >>> Did you see a message in the logs confirming that the setting has been >>> updated? It would be interesting to see the output of hot threads[1] to see >>> what your node is doing. >>> >>> [1] http://www.elasticsearch.org/guide/en/elasticsearch/ >>> reference/current/cluster-nodes-hot-threads.html >>> >>> >>> On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw >>> <ebradsh...@gmail.com>wrote: >>> >>>> Yes. I have run max_num_segments=1 every time. >>>> >>>> >>>> On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick < >>>> michael.s...@serenesoftware.com> wrote: >>>> >>>>> Have you tried max_num_segments=1 on your optimize? >>>>> >>>>> On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw < >>>>> ebradsh...@gmail.com> wrote: >>>>> >>>>>> Any thoughts on this? I've run optimize several more times, and the >>>>>> number of segments falls each time, but I'm still over 1000 segments per >>>>>> shard. Has anyone else run into something similar? >>>>>> >>>>>> >>>>>> On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: >>>>>>> >>>>>>> OK. Optimize finally returned, so I suppose something was happening >>>>>>> in the background, but I'm still seeing over 6500 segments. Even after >>>>>>> setting max_num_segments=5. Does this seem right? Queries are a little >>>>>>> faster (350-400ms) but still not great. Bigdesk is still showing a fair >>>>>>> amount of file IO. >>>>>>> >>>>>>> On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node >>>>>>>> cluster, each with 64G of ram, with 24G allocated to Elasticsearch on >>>>>>>> each. I've batch loaded approximately 86 million documents into a >>>>>>>> single >>>>>>>> index (4 shards) and have started benchmarking cross_field/multi_match >>>>>>>> queries on them. The index has one replica and takes up a total of >>>>>>>> 111G. >>>>>>>> I've run several batches of warming queries, but queries are not as >>>>>>>> fast as >>>>>>>> I had hoped, approximately 400-500ms each. Given that *top *(on >>>>>>>> Centos) shows 5-8 GB of free memory on each server, I would assume >>>>>>>> that the >>>>>>>> entire index has been paged into memory (I had worried about disk >>>>>>>> performance previously, as we are working in a virtualized >>>>>>>> environment). >>>>>>>> >>>>>>>> A stats query on the index in questions shows that the index is >>>>>>>> composed of > 7000 segments. This seemed high to me, but maybe it's >>>>>>>> appropriate. Regardless, I dispatched an optimize command, but I am >>>>>>>> not >>>>>>>> seeing any progress and the command has not returned. Current merges >>>>>>>> remains at zero, and the segment count is not changing. Checking out >>>>>>>> hot >>>>>>>> threads in ElasticHQ, I initially saw an optimize call in the stack >>>>>>>> that >>>>>>>> was blocked on a waitForMerge call. This however has disappeared, and >>>>>>>> I'm >>>>>>>> seeing no evidence that the optimize is occuring. >>>>>>>> >>>>>>>> Does any of this seem out of the norm or unusual? Has anyone else >>>>>>>> had similar issues. This is the second time I have tried to optimize >>>>>>>> an >>>>>>>> index since upgrading. I've gotten the same result both time. >>>>>>>> >>>>>>>> Thanks in advance for any help/tips! >>>>>>>> >>>>>>>> - Elliott >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/5391291f- >>>>>> 5c5e-4088-a1f2-93272beef0bb%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>> topic/elasticsearch/kqTRRADQBwc/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/CAP8axnD7BUziGct2%3Db% >>>>> 3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW% >>>> 3Demc0JTouT9%2BBeUk_A%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Adrien Grand >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>> topic/elasticsearch/kqTRRADQBwc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8% >>> 2BKDsg%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4D577J%2B0NvmUwq7Kh049ZBW2hu8vGnzrZFefwazc1Eow%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
