Re: Queue capacity and EsRejectedExecutionException leads to loss of data
I think that adding a comment into the existing issue would be fine. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 26 février 2014 à 17:00:12, Thomas (thomas.bo...@gmail.com) a écrit: Thanks David, So this is a rabbitMQRiver issue, is there a need to open a separate issue? (Never done the procedure, will look this one) Thomas On Wednesday, 26 February 2014 15:48:55 UTC+2, Thomas wrote: Hi, We have installed the RabbitMQ river plugin to pull data from our Queue and adding them to ES. The thing is that at some point we are receiving the following exception and we have as a result to lose data. [1775]: index [events-idx], type [click], id [3f6e4604146b435aabcf4ea5a493fd32], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]] We have changed the configuration of queue size to 1000 and the problem disappeared. My question is that is there any configuration/way to tell ES to instead of throwing this exception and discarding the document to wait for available resources (with the corresponding performance impact)? Thanks Thomas -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03dcb0ea-2b6a-478b-b678-f52ecbc09298%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.530e1318.440badfc.5e46%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Re: Queue capacity and EsRejectedExecutionException leads to loss of data
Thanks David, So this is a rabbitMQRiver issue, is there a need to open a separate issue? (Never done the procedure, will look this one) Thomas On Wednesday, 26 February 2014 15:48:55 UTC+2, Thomas wrote: > > Hi, > > We have installed the RabbitMQ river plugin to pull data from our Queue > and adding them to ES. The thing is that at some point we are receiving the > following exception and we have as a result to *lose data*. > > [1775]: index [events-idx], type [click], id >> [3f6e4604146b435aabcf4ea5a493fd32], message >> [EsRejectedExecutionException[rejected execution (queue capacity 50) on >> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]] > > > We have changed the configuration of queue size to 1000 and the problem > disappeared. > > My question is that is there any configuration/way to tell ES to instead > of throwing this exception and discarding the document to wait for > available resources (with the corresponding performance impact)? > > Thanks > > Thomas > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03dcb0ea-2b6a-478b-b678-f52ecbc09298%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Queue capacity and EsRejectedExecutionException leads to loss of data
I think you should open an issue. It's to me somehow related to this issue: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/47 and its pending PR: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/pull/48 I think we should have a new option to let the user decides if if want to ack messages in case of failure or leave them in queue. My 2 cents -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr Le 26 février 2014 à 14:49:00, Thomas (thomas.bo...@gmail.com) a écrit: Hi, We have installed the RabbitMQ river plugin to pull data from our Queue and adding them to ES. The thing is that at some point we are receiving the following exception and we have as a result to lose data. [1775]: index [events-idx], type [click], id [3f6e4604146b435aabcf4ea5a493fd32], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]] We have changed the configuration of queue size to 1000 and the problem disappeared. My question is that is there any configuration/way to tell ES to instead of throwing this exception and discarding the document to wait for available resources (with the corresponding performance impact)? Thanks Thomas -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e11ce4e-26d4-4a53-a53e-0ba89bde4605%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.530dfa41.1a70.5e46%40MacBook-Air-de-David.local. For more options, visit https://groups.google.com/groups/opt_out.
Queue capacity and EsRejectedExecutionException leads to loss of data
Hi, We have installed the RabbitMQ river plugin to pull data from our Queue and adding them to ES. The thing is that at some point we are receiving the following exception and we have as a result to *lose data*. [1775]: index [events-idx], type [click], id > [3f6e4604146b435aabcf4ea5a493fd32], message > [EsRejectedExecutionException[rejected execution (queue capacity 50) on > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]] We have changed the configuration of queue size to 1000 and the problem disappeared. My question is that is there any configuration/way to tell ES to instead of throwing this exception and discarding the document to wait for available resources (with the corresponding performance impact)? Thanks Thomas -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e11ce4e-26d4-4a53-a53e-0ba89bde4605%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.