When we were unspooling from disk after a drop to DA mode, we would definitely hit 100% cpu utilization on the core that was doing the work. An important point to note is that we were also probably still intaking around 60,000 messages a second on that rule -while- this was occurring. It could perhaps be a situation specific to unspooling from disk while still intaking a largish volume of traffic?
Brian On Thu, Nov 1, 2012 at 12:29 PM, Rainer Gerhards <[email protected]> wrote: > > >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Brian Knox >> Sent: Wednesday, October 31, 2012 6:12 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or redesign? >> >> I have two initial thoughts from experience concerning heavy use of >> the queue system in high traffic (6+ billion log lines a day) >> environments - specifically concerning DA queues. >> >> 1. The queues for the most part were very reliable >> 2. The performance unspooling from a "fail" state (defined as "we had >> to spool to disk") was sub optimal. Using fast io subsystems also >> seemed to have little impact on the unspool speed - it seemed stuck at >> around 50 to 60k messages a second in our environment, and seemed to >> be more CPU bound than IO bound. > > This is pretty interesting to me. I'd always thought that the disk queue > system is heavily io bound, but uses very low CPU. I think I even tested > this, but not sure any longer. Performance was of no concern for the initial > design. > > Rainer >> So, in our production we always tried to set the in memory portion of >> the queues very high - the disk backed portion was there in case of a >> disaster and we daily prayed we didn't end up having to fail over to >> them. We lived in fear of finding ourselves in a situation where we >> could not unspool fast enough to catch back up. >> >> The queue performance was a definite limiting factor for us. It was >> probably are biggest pain point with rsyslog. >> >> I don't say this to trivialize the complexity of the problem - and to >> be clear, we rarely had -reliability- issues with the queues. >> >> Brian >> >> >> On Wed, Oct 31, 2012 at 12:55 PM, Rainer Gerhards >> <[email protected]> wrote: >> > Hi all, >> > >> > There is the dangling issue that rsyslog has grown out of its current >> queue subsystem. I am currently considering a refactoring or a complete >> redesign. I initially wanted to write a large blog post with all >> details and ideas, but have now opted to split this in a couple of >> parts - both because I have problems to find time to do the "big one" >> at once; and also it probably is smarter to get feedback asap. >> > >> > So here is the initial part: >> > >> > http://blog.gerhards.net/2012/10/rsyslog-disk-queues-refactor-or.html >> > >> > This will get anyone interested in the queue subsystem a broad >> understanding of how it works - and why. Please share any concerns you >> have about the current system as well as wishes/suggestions on what >> should improve. Deeply technical information is fine, actually >> appreciated. >> > >> > I intend to let the discussion run and write the other parts of the >> blog series when "events warrant it" ;) Due to other projects, I can >> probably not discuss 10 hours a day, but will try to be as active as >> possible (which hopefully means "much"). The intent is to come up with >> a solution that will be good for the next five years to come... >> > >> > Thanks, >> > Rainer >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if you DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
