Hello, We recently set up a new host with graylog 1.2.2 and elasticsearch 1.7.3 from the centos repos available. Every 3 or 4 days graylog will lose it's connection to elasticsearch and stop outputting messages to the database. The messages are kept in the disk buffer and will be put into elasticsearch upon restarting the graylog-server process.
The graylog web interface reports everything is green and curl reqests to elasticsearch show a-ok. I found the following message in the graylog-server log but it does not say why... 2015-11-24T08:52:37.389-08:00 WARN [BlockingBatchedESOutput] Error while waiting for healthy Elasticsearch cluster. Not flushing. java.util.concurrent.TimeoutException: Elasticsearch cluster didn't get healthy within timeout at org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:174) at org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:179) at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:112) at org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105) at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Our ES is configured to use 30G of ram and graylog server is configured for 4GB. The host has 74GB total. Let me know if i can provide any more info. Anyone experience this? Our old setup did not have this problem... Thanks, Josh -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/ad0c172a-cfb5-493f-bd71-3eb021c7a51a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
