I did find the shutdown request in the syslog:
Aug 23 16:49:01 medisafelog2 kernel: [3361057.489168] hv_utils: Shutdown
request received - graceful shutdown initiated
yet i have no idea who or what initiated it... how can i dig in?
On Sunday, August 24, 2014 12:01:50 PM UTC+3, Eitan Vesely wrote:
>
>
> Thanks Mark,
>
> auth.log doesnt show any login or sudo at the time of the elastic
> stopping...
> nothing else is running on that machine - it is a dedicated ES server.
>
> what i did find in the auth log is that someone is trying to hack into the
> system, yet i dont see how it got to do with elastic stopping?
>
> On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote:
>>
>> Something is stopping the service.
>>
>> If you are on linux check the auth log, if anyone is using sudo to stop
>> it then you will see that logged. Otherwise, what else runs on the machine?
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 24 August 2014 06:15, Eitan Vesely <eita...@gmail.com> wrote:
>>
>>> Hi Guys,
>>> i've installed ES a month ago and its working just fine.
>>>
>>> today, for some reason, ES just went down for no visible reason:
>>>
>>> here is what i see in the log file :
>>>
>>> [2014-08-23 16:47:11,272][DEBUG][action.search.type ] [Plunderer]
>>> [g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
>>> Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
>>> lastShard [true]
>>> org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][1]:
>>> from[-1],size[-1]: Parse Failure [Failed to parse source
>>> [{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
>>> at
>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
>>> at
>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
>>> at
>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
>>> at
>>> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
>>> at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
>>> at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
>>> at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
>>> Facet [0]: (key) field [@timestamp] not found
>>> at
>>> org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
>>> at
>>> org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
>>> at
>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
>>> ... 9 more
>>> [2014-08-23 16:47:11,273][DEBUG][action.search.type ] [Plunderer]
>>> [g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
>>> Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
>>> org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][0]:
>>> from[-1],size[-1]: Parse Failure [Failed to parse source
>>> [{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
>>> at
>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
>>> at
>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
>>> at
>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
>>> at
>>> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
>>> at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
>>> at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
>>> at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
>>> Facet [0]: (key) field [@timestamp] not found
>>> at
>>> org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
>>> at
>>> org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
>>> at
>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
>>> ... 9 more
>>> [2014-08-23 16:49:05,401][INFO ][node ] [Plunderer]
>>> stopping ...
>>> [2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
>>> stopped
>>> [2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
>>> closing ...
>>> [2014-08-23 16:49:06,772][INFO ][node ] [Plunderer]
>>> closed
>>> [2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to
>>> lock JVM memory (ENOMEM). This can result in part of the JVM being swapped
>>> out. Increase RLIMIT_MEMLOCK or run elasticsearch as root.
>>> [2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
>>> Abdol] version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
>>> [2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
>>> Abdol] initializing ...
>>> [2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet
>>> Abdol] loaded [], sites [kopf]
>>> [2014-08-23 20:02:14,692][INFO ][node ] [Ahmet
>>> Abdol] initialized
>>>
>>> any ideas??
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
