On 05/17/2018 05:52 AM, Brian Knox wrote:
To my knowledge, Rich is correct. This also would explain a case we hit maybe every couple of months, where rsyslog very quickly duplicates some messages it is sending to elasticsearch. I would assume this would be a case where a batch is submitted, only some of the messages are rejected, and rsyslog then duplicates messages trying to send the batch over and over again.
You can confirm this by monitoring the bulk index thread pool https://www.elastic.co/guide/en/elasticsearch/reference/2.4/cat-thread-pool.html to see if you are getting bulk rejections.
On Thu, May 17, 2018 at 12:08 AM David Lang <da...@lang.hm <mailto:da...@lang.hm>> wrote:On Wed, 16 May 2018, Rich Megginson wrote: > On 05/16/2018 05:58 PM, David Lang wrote: >> there's no need to add this extra complexity (multiple rulesets and queues) >> >> What should be happening (on any output module) is: >> >> submit a batch. >> If rejected with a soft error, retry/suspend the output > > retry of the entire batch? see below > >> if batch-size=1 and a hard error, send to errorfile >> if rejected with a hard error resubmit half of the batch > > But what if 90% of the batch was successfully added? Then you are needlessly > resubmitting many of the records in the batch. when submitting batches, you get a success/fail for the batch as a whole (for 99% of things that actually allow you to insert in batches), so you don't know what message failed. This is a database transaction (again, in most cases), so if a batch fails, all you can do is bisect to figure out what message fails. If the endpoint is inserting some of the messages from a batch that fails, that's usually a bad thing. now, if ES batch mode isn't an ACID transaction and it accepts some messages and then tells you which ones failed, then you can mark the ones accepted as done and just retry the ones that fail. But there's still no need for a separate ruleset and queue. In Rsyslog, if an output cannot accept a message and there's reason to think that it will in the future, then you suspend that output and try again later. If you have reason to believe that the message is never going to be able to be delivered, then you need to fail the message or you will be stuck forever. This is what the error output was made for. > If using the "index" (default) bulk type, this causes duplicate records to be > added. > If using the "create" type (and you have assigned a unique _id), you will get > back many 409 Duplicate errors. > This causes problems - we know because this is how the fluentd plugin used to > work, which is why we had to change it. > > https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_monitoring_individual_nodes.html#_threadpool_section > "Bulk Rejections" > "It is much better to handle queuing in your application by gracefully > handling the back pressure from a full queue. When you receive bulk > rejections, you should take these steps: > > Pause the import thread for 3–5 seconds. > Extract the rejected actions from the bulk response, since it is probable > that many of the actions were successful. The bulk response will tell you > which succeeded and which were rejected. > Send a new bulk request with just the rejected actions. > Repeat from step 1 if rejections are encountered again. > > Using this procedure, your code naturally adapts to the load of your cluster > and naturally backs off. > " Does it really accept some and reject some in a random manner? or is it a matter of accepting the first X and rejecting any after that point? The first is easier to deal with. Batch mode was created to be able to more efficiently process messages that are inserted into databases, we then found that the reduced queue congestion was a significant advantage in itself. But unless you have a queue just for the ES action, doing queue manipulation isn't possible, all you can do is succeed or fail, and if you fail, the retry logic will kick in. Rainer is going to need to comment on this. David Lang > >> repeat >> >> all that should be needed is to add tests into omelasticsearch to detect >> the soft errors and turn them into retries (or suspend the output as >> appropriate) >> >> David Lang > > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
