On Fri, 21 Aug 2015, Otis Gospodnetić wrote:
Hi,
This sounds like something that should be om-specific. What Radu is
suggesting would definitely help with ES, but may not be relevant for other
output targets.
What I think is overlooked here is the ES side - more specifically ES and
searches that ES has to handle. If we don't care about maxing out ES and
just pushing data in it as fast as it arrives, then how
rsyslog/omelasticsearch works today makes sense. But this approach if
focused on ingestion and ignores how this can hurt ES's ability to handle
queries in a timely manner. Exposing controls Radu suggested would help
people avoid this problem. I know David would like to see numbers :) I
love numbers, too, but I'm not sure if we'll have the time to provide them
:( That said, we work with ES 24/7 and have been doing that for years
(many hundreds of ES deployments under our belt by now), so I am hoping
somebody will trust us this option would be great to have in
omelasticsearch. :)
I think that this really should be addressed on the ElasticSearch side of
things.
This really shouldn't be a numerical limit thing.
What is ideal is that if ES is lightly loaded, things get pushed into ES with
the minimum latency. But if ES is more heavily loaded, batch things up.
The right way to do this (as I said in another discussion) is for ES to have a
way to prioritize searches over inputting new data. That way as the load climbs,
the rate of processing new inserts will slow and inserts will get batched more.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
