Hello,

We recently set up a new host with graylog 1.2.2 and elasticsearch 1.7.3 
from the centos repos available. Every 3 or 4 days graylog will lose it's 
connection to elasticsearch and stop outputting messages to the database. 
The messages are kept in the disk buffer and will be put into elasticsearch 
upon restarting the graylog-server process. 

The graylog web interface reports everything is green and curl reqests to 
elasticsearch show a-ok. I found the following message in the 
graylog-server log but it does not say why... 

2015-11-24T08:52:37.389-08:00 WARN  [BlockingBatchedESOutput] Error while 
waiting for healthy Elasticsearch cluster. Not flushing.

java.util.concurrent.TimeoutException: Elasticsearch cluster didn't get 
healthy within timeout

at 
org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:174)

at 
org.graylog2.indexer.cluster.Cluster.waitForConnectedAndHealthy(Cluster.java:179)

at 
org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:112)

at 
org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:105)

at 
org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189)

at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)


Our ES is configured to use 30G of ram and graylog server is configured for 
4GB. The host has 74GB total.


Let me know if i can provide any more info. Anyone experience this? Our old 
setup did not have this problem...


Thanks,

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ad0c172a-cfb5-493f-bd71-3eb021c7a51a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to