Any other thoughts on this? Would 1500 segments per shard be significantly impacting performance? Have you guys noticed this behavior elsewhere?
Thanks. On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote: > > Adrian, > > I ran the following command: > > curl -XPUT http://localhost:9200/_settings -d > '{"indices.store.throttle.max_bytes_per_sec" : "10gb"}' > > and received a { "acknowledged" : "true" } response. The logs showed > "cluster state updated". > > I did have to close my index prior to changing the setting and reopen > afterward. > > > I've since began another optimize, but again it doesn't look like much is > happening. The optimize isn't returning and the total CPU usage on every > node is holding at about 2% of a single core. I would copy a hot_threads > stack trace, but I'm unfortunately on a closed network and this isn't > possible. I can tell you that refreshes of hot_threads show vary little > happening. The occasional [merge] thread (always in a > LinkedTransferQueue.awaitMatch() state) or [optimize] (doing nothing on a > waitForMerge() call) thread shows up, but it's always consuming 0-1% CPU. > It sure feels like something isn't right. Any thoughts? > > > > > On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand < > adrien.gr...@elasticsearch.com> wrote: > >> Did you see a message in the logs confirming that the setting has been >> updated? It would be interesting to see the output of hot threads[1] to see >> what your node is doing. >> >> [1] >> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html >> >> >> On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <ebradsh...@gmail.com>wrote: >> >>> Yes. I have run max_num_segments=1 every time. >>> >>> >>> On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick < >>> michael.s...@serenesoftware.com> wrote: >>> >>>> Have you tried max_num_segments=1 on your optimize? >>>> >>>> On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <ebradsh...@gmail.com >>>> > wrote: >>>> >>>>> Any thoughts on this? I've run optimize several more times, and the >>>>> number of segments falls each time, but I'm still over 1000 segments per >>>>> shard. Has anyone else run into something similar? >>>>> >>>>> >>>>> On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott Bradshaw wrote: >>>>>> >>>>>> OK. Optimize finally returned, so I suppose something was happening >>>>>> in the background, but I'm still seeing over 6500 segments. Even after >>>>>> setting max_num_segments=5. Does this seem right? Queries are a little >>>>>> faster (350-400ms) but still not great. Bigdesk is still showing a fair >>>>>> amount of file IO. >>>>>> >>>>>> On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott Bradshaw wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I've recently upgraded to Elasticsearch 1.1.0. I've got a 4 node >>>>>>> cluster, each with 64G of ram, with 24G allocated to Elasticsearch on >>>>>>> each. I've batch loaded approximately 86 million documents into a >>>>>>> single >>>>>>> index (4 shards) and have started benchmarking cross_field/multi_match >>>>>>> queries on them. The index has one replica and takes up a total of >>>>>>> 111G. >>>>>>> I've run several batches of warming queries, but queries are not as >>>>>>> fast as >>>>>>> I had hoped, approximately 400-500ms each. Given that *top *(on >>>>>>> Centos) shows 5-8 GB of free memory on each server, I would assume that >>>>>>> the >>>>>>> entire index has been paged into memory (I had worried about disk >>>>>>> performance previously, as we are working in a virtualized environment). >>>>>>> >>>>>>> A stats query on the index in questions shows that the index is >>>>>>> composed of > 7000 segments. This seemed high to me, but maybe it's >>>>>>> appropriate. Regardless, I dispatched an optimize command, but I am >>>>>>> not >>>>>>> seeing any progress and the command has not returned. Current merges >>>>>>> remains at zero, and the segment count is not changing. Checking out >>>>>>> hot >>>>>>> threads in ElasticHQ, I initially saw an optimize call in the stack >>>>>>> that >>>>>>> was blocked on a waitForMerge call. This however has disappeared, and >>>>>>> I'm >>>>>>> seeing no evidence that the optimize is occuring. >>>>>>> >>>>>>> Does any of this seem out of the norm or unusual? Has anyone else >>>>>>> had similar issues. This is the second time I have tried to optimize >>>>>>> an >>>>>>> index since upgrading. I've gotten the same result both time. >>>>>>> >>>>>>> Thanks in advance for any help/tips! >>>>>>> >>>>>>> - Elliott >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Adrien Grand >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
