> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Brian Knox > Sent: Wednesday, October 31, 2012 6:12 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or redesign? > > I have two initial thoughts from experience concerning heavy use of > the queue system in high traffic (6+ billion log lines a day) > environments - specifically concerning DA queues. > > 1. The queues for the most part were very reliable > 2. The performance unspooling from a "fail" state (defined as "we had > to spool to disk") was sub optimal. Using fast io subsystems also > seemed to have little impact on the unspool speed - it seemed stuck at > around 50 to 60k messages a second in our environment, and seemed to > be more CPU bound than IO bound.
This is pretty interesting to me. I'd always thought that the disk queue system is heavily io bound, but uses very low CPU. I think I even tested this, but not sure any longer. Performance was of no concern for the initial design. Rainer > So, in our production we always tried to set the in memory portion of > the queues very high - the disk backed portion was there in case of a > disaster and we daily prayed we didn't end up having to fail over to > them. We lived in fear of finding ourselves in a situation where we > could not unspool fast enough to catch back up. > > The queue performance was a definite limiting factor for us. It was > probably are biggest pain point with rsyslog. > > I don't say this to trivialize the complexity of the problem - and to > be clear, we rarely had -reliability- issues with the queues. > > Brian > > > On Wed, Oct 31, 2012 at 12:55 PM, Rainer Gerhards > <[email protected]> wrote: > > Hi all, > > > > There is the dangling issue that rsyslog has grown out of its current > queue subsystem. I am currently considering a refactoring or a complete > redesign. I initially wanted to write a large blog post with all > details and ideas, but have now opted to split this in a couple of > parts - both because I have problems to find time to do the "big one" > at once; and also it probably is smarter to get feedback asap. > > > > So here is the initial part: > > > > http://blog.gerhards.net/2012/10/rsyslog-disk-queues-refactor-or.html > > > > This will get anyone interested in the queue subsystem a broad > understanding of how it works - and why. Please share any concerns you > have about the current system as well as wishes/suggestions on what > should improve. Deeply technical information is fine, actually > appreciated. > > > > I intend to let the discussion run and write the other parts of the > blog series when "events warrant it" ;) Due to other projects, I can > probably not discuss 10 hours a day, but will try to be as active as > possible (which hopefully means "much"). The intent is to come up with > a solution that will be good for the next five years to come... > > > > Thanks, > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
