Hello rsyslog users :) We've seen a problem that is similar to the one reported here: http://www.gossamer-threads.com/lists/rsyslog/users/17550 While that looks like a bug, ours seems like a design issue.
Basically we see bulks of one document all over the place. Not 100% what's the root cause, but I'm thinking: if you have many machines with rsyslog installed that send logs to Elasticsearch, but most of them send little logs, they would never get enough messages in the queue to push in large batches. Unless you add a slowdown, in which case you restrict rsyslog's ability to push data when it's under load. If you have 10K docs/s coming in 1 doc batches (say, from 10K machines), there's a lot of unnecessary load on ES. Sure, if ES is overloaded things will get better (as documents will add up in queues, resulting in bigger batches) but even then I'd imagine things will look quite inefficient. Plus, I'd like to avoid ES being overloaded in the first place. The solution, in my mind, was to add two options: - one that says "if you don't have at least N items in the bulk, wait a bit until you have" - one that overrides it saying "if M seconds passed since the last bulk, send the bulk anyway" Now the big questions: - is this possible? where would one apply such a change? - would it have a significant impact on the performance of outputs that work well with the current design? Like omfwd, where the receiving end wouldn't care how many docs it receives I imagine - if it does have a significant impact, can we restrict such a change to omelasticsearch, or does it have to go under rsyslog's core (in the way it handles queues)? - do you see better solutions? Thanks and best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
