Sometimes that happens when the new node starts up via monit or other automated thing before the old node is fully shutdown. I'd suggest shutting down the node and verify it's done via ps before allowing the new node to start up. In the case of monit if the check is hitting the http port then it'll think it's down before it actually fully quits.
On Tuesday, March 24, 2015 at 1:56:54 PM UTC-4, Yogesh wrote: > > Thanks a lot mjdude! It does seem like it attached to the wrong data > directory. > In elasticsearch/data/tool/nodes there are two 0 and 1. My data is in 0 > but node stats shows the data directory as elasticsearch/data/tool/nodes/1. > Now, how do I change this? > > On Tue, Mar 24, 2015 at 11:02 PM, <mjd...@gmail.com <javascript:>> wrote: > >> When it restarted did it attach to the wrong data directory? Take a look >> at _nodes/_local/stats?pretty and check the 'data' directory location. Has >> the cluster recovered after the restart? Check _cluster/health?pretty as >> well. >> >> On Tuesday, March 24, 2015 at 1:01:52 PM UTC-4, Yogesh wrote: >>> >>> Thanks Joel and mjdude. What I mean is that ES is using 99% of the heap >>> memory (I think since Marvel showed memory as 1 GB which corresponds to the >>> heap, my RAM is 50GB) >>> I've increased the ES_HEAP_SIZE to 10g. But another problem has appeared >>> and I'm freaked out because of it! >>> >>> So, after restarting my ES (curl shutdown) to increase the heap, the >>> Marvel has stopped showing me my data (it still shows that disk memory is >>> lower so that means the data is still on disk) and upon searching Sense >>> shows "IndexMissingException[[my_new_twitter_river] missing]" >>> >>> Why is this happening?!?! >>> >>> On Mon, Mar 23, 2015 at 9:25 PM, <mjd...@gmail.com> wrote: >>> >>>> Are you saying JVM is using 99% of the system memory or 99% of the >>>> heap? If it's 99% of the available heap that's bad and you will have >>>> cluster instability. I suggest increasing your JVM heap size if you can, >>>> I >>>> can't find it right now but I remember a blog post that used twitter as a >>>> benchmark and they also could get to ~50M documents with the default 1G >>>> heap. >>>> >>>> On Sunday, March 22, 2015 at 3:30:57 AM UTC-4, Yogesh wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have set up elasticsearch on one node and am using the Twitter river >>>>> to index tweets. It has been going fine with almost 50M tweets indexed so >>>>> far in 13 days. >>>>> When I started indexing, the JVM usage (observed via Marvel) hovered >>>>> between 10-20%, then started remaining around 30-40% but for the past 3-4 >>>>> days it has continuously been above 90%, reaching 99% at times! >>>>> I restarted elasticsearch thinking it might get resolved but as soon >>>>> as I switched it back on, the JVM usage went back to 90%. >>>>> >>>>> Why is this happening and how can I remedy it? (The JVM memory is the >>>>> default 990.75MB) >>>>> >>>>> Thanks >>>>> Yogesh >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>> topic/elasticsearch/kMSwBqpe2N4/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> elasticsearc...@googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/0f990291-fa20-4bba-881f-3f378985c8c9% >>>> 40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/0f990291-fa20-4bba-881f-3f378985c8c9%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/kMSwBqpe2N4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/057e53f7-3207-4df9-bbc4-80cea03e189f%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/057e53f7-3207-4df9-bbc4-80cea03e189f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a3ca6aa0-f3d9-4067-933c-1a2eb0a15075%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
