Hi there I just did a simple search on 30 days of data and managed to trigger the following ES error
[2016-06-01 00:12:53,525][WARN ][indices.breaker.fielddata] [fielddata] New used memory 11273780309 [10.4gb] for data of [message] would be larger than configured breaker: 10857952051 [10.1gb], breaking According to what I can google, this means that ES would have had to allocate more resources than available to fulfil it, and that condition somehow triggers an epic fail: either ES becomes unresponsive or graylog-server does - I can't tell the difference. All I know is right now I have messages going into graylog and nothing coming out. Within a minute, things went bad to worse, suddenly I'm getting shard errors (first shard errors in ages - definitely related) [2016-06-01 00:21:32,860][WARN ][indices.cluster ] [fantail] [[graylog_488][0]] marking and sending shard failed due to [engine failure, reason [already closed by tragic event on the index writer]] [graylog_488][[graylog_488][0]] ShardNotFoundException[no such shard] at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:197) [2016-06-01 00:21:32,962][WARN ][cluster.action.shard ] [fantail] [graylog_488][0] received shard failed for target shard [[graylog_488][0], node[Tjzmk9cFRuCke6JEuomb4g], [P], v[2], s[STARTED], a[id=dgyATFPBQAywkydc2mxmPw]], indexUUID [jxF7U5fESqOzJu9CSDF3WA], message [engine failure, reason [already closed by tragic event on the index writer]], failure [OutOfMemoryError[Java heap space]] [2016-06-01 00:21:32,974][WARN ][cluster.action.shard ] [fantail] [graylog_488][0] received shard failed for target shard [[graylog_488][0], node[Tjzmk9cFRuCke6JEuomb4g], [P], v[2], s[STARTED], a[id=dgyATFPBQAywkydc2mxmPw]], indexUUID [jxF7U5fESqOzJu9CSDF3WA], message [master {fantail}{Tjzmk9cFRuCke6JEuomb4g}{127.0.0.1}{127.0.0.1:9300} marked shard as started, but shard has previous failed. resending shard failure.] [2016-06-01 00:21:33,182][INFO ][cluster.routing.allocation] [fantail] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[graylog_488][0], [graylog_488][0]] ...]). Restarting graylog-server and ES (and cleaning up...) will solve this - but this is lame. graylog is an end-user tool that *by design* will have people doing actions that - on occasion - are beyond the reach of the backend: there has to be some way this could be handled better. The ES people seem to think this is a case of "you're doing it wrong", but graylog isn't some programmed frontend where every ES call is tightly managed - it's something that is meant to be used to "play" with data. Basically all I did was take a previous search that worked and asked it to re-run with an hourly graph instead of daily - enough to tip it over the edge. This will happen time and time again - so causing service outages is an acceptable outcome? How are others dealing with this? Could graylog capture the ES error and mitigate (somehow)? I for one should have shut everything down before that "breaker" error turned into the "shard" error. This is graylog-server-2.0.2/elasticsearch-2.3.3 under CentOS-7 Thanks Jason -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/b7a7b095-3b6d-47fb-8bb0-bc62b8b67011%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.