I did some basic checks with the Count API and noticed: curl -XGET 'http://localhost:9200/_all/_count' -d '{"query":{"term":{"source":"example.com"}}}' {"count":14279142,"_shards":{"total":1600,"successful":1587,"failed":13,"failures":[{"index":"graylog_536","shard":2,"status":429,"reason":"BroadcastShardOperationFailedException[[graylog_536][2] ]; nested: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@5f1f5aa1]; "},{"index":"graylog_569","shard":2,"status":429,"reason":"BroadcastShardOperationFailedException[[graylog_569][2] ]; nested: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@e148fb1]; "} ...
And of course /var/log/graylog/elasticsearch/current showed the same: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) So only some of the indices were scanned. I changed elasticsearch.yml to include this setting: threadpool.search.type: cached and that stopped the errors. Histograms show things properly now. Counts are correct in the UI. I do think it is a bug that the Graylog UI showed "found XXX messages in 1,234ms searched in YYY indices", when in fact it silently failed to search all those indices. On Friday, August 14, 2015 at 11:48:23 AM UTC-6, Jesse Skrivseth wrote: > > Obvious they should change. ;) > > But the problem is that they are all over the place. If I do an all-time > search for something simple, like source:xxx, then do any type of > histogram, every time that histogram refreshes the whole graph changes, > even messages from days/weeks ago, by huge magnitudes (10 million messages > from 14 days ago suddenly becomes 20,000). > > I am baffled. This occurs in two instances with totally different data > sets. One is running 1.1.3 and the other is running 1.1.6. We do have a > process that uses the Elasticsearch S3 plugin to archive closed indices and > deletes them using elasticsearch API directly. Maybe that's somehow causing > problems? > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/203dbb4f-2bae-40a1-bac8-986f470380f7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
