Hello all, I’m getting failed nodes when running searches and I’m hoping someone can point me in the right direction. I have indices created per day to store messages. The pattern is pretty straight forward: the index for January 1 is "messages_20140101", for January 2 is "messages_20140102" and so on. Each index is created against a template that specifies 20 shards. A full year will give 365 indices * 20 shards = 7300 nodes. I have recently upgraded to ES 1.0.
When I search for all messages in a year (either using an alias or specifying “messages_2013*”), I get many failed nodes. The reason given is: “EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@651b8924]”). The more often I search, the fewer failed nodes I get (probably caching in ES) but I can’t get down to 0 failed nodes. I’m using ES for analytics so the document counts coming back have to be accurate. The aggregate counts will change depending on the number of node failures. We use the Java API to create a local node to index and search the documents. However, we also see the issue if we use the URL search API on port 9200. If I restrict the search for 30 days then I do not see any failures (it’s under 1000 nodes so as expected). However, it is a pretty common use case for our customers to search messages spanning an entire year. Any suggestions on how I can prevent these failures? Thank you for your help! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9bf6d3bb-34e5-44c4-8d76-24f868d283a0%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
