Thanks Mark, auth.log doesnt show any login or sudo at the time of the elastic stopping... nothing else is running on that machine - it is a dedicated ES server.
what i did find in the auth log is that someone is trying to hack into the system, yet i dont see how it got to do with elastic stopping? On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote: > > Something is stopping the service. > > If you are on linux check the auth log, if anyone is using sudo to stop it > then you will see that logged. Otherwise, what else runs on the machine? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: ma...@campaignmonitor.com <javascript:> > web: www.campaignmonitor.com > > > On 24 August 2014 06:15, Eitan Vesely <eita...@gmail.com <javascript:>> > wrote: > >> Hi Guys, >> i've installed ES a month ago and its working just fine. >> >> today, for some reason, ES just went down for no visible reason: >> >> here is what i see in the log file : >> >> [2014-08-23 16:47:11,272][DEBUG][action.search.type ] [Plunderer] >> [g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]: >> Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad] >> lastShard [true] >> org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][1]: >> from[-1],size[-1]: Parse Failure [Failed to parse source >> [{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]] >> at >> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649) >> at >> org.elasticsearch.search.SearchService.createContext(SearchService.java:511) >> at >> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483) >> at >> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252) >> at >> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206) >> at >> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203) >> at >> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: >> Facet [0]: (key) field [@timestamp] not found >> at >> org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160) >> at >> org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93) >> at >> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633) >> ... 9 more >> [2014-08-23 16:47:11,273][DEBUG][action.search.type ] [Plunderer] >> [g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]: >> Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad] >> org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][0]: >> from[-1],size[-1]: Parse Failure [Failed to parse source >> [{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]] >> at >> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649) >> at >> org.elasticsearch.search.SearchService.createContext(SearchService.java:511) >> at >> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483) >> at >> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252) >> at >> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206) >> at >> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203) >> at >> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: >> Facet [0]: (key) field [@timestamp] not found >> at >> org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160) >> at >> org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93) >> at >> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633) >> ... 9 more >> [2014-08-23 16:49:05,401][INFO ][node ] [Plunderer] >> stopping ... >> [2014-08-23 16:49:06,737][INFO ][node ] [Plunderer] >> stopped >> [2014-08-23 16:49:06,737][INFO ][node ] [Plunderer] >> closing ... >> [2014-08-23 16:49:06,772][INFO ][node ] [Plunderer] >> closed >> [2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to >> lock JVM memory (ENOMEM). This can result in part of the JVM being swapped >> out. Increase RLIMIT_MEMLOCK or run elasticsearch as root. >> [2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol] >> version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z] >> [2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol] >> initializing ... >> [2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet Abdol] >> loaded [], sites [kopf] >> [2014-08-23 20:02:14,692][INFO ][node ] [Ahmet Abdol] >> initialized >> >> any ideas?? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d87b8ca2-8f11-4f5c-b0f1-d661d5bf2f3b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.