> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Brian Knox > Sent: Thursday, November 01, 2012 6:28 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or redesign? > > When we were unspooling from disk after a drop to DA mode, we would > definitely hit 100% cpu utilization on the core that was doing the > work. An important point to note is that we were also probably still > intaking around 60,000 messages a second on that rule -while- this was > occurring. It could perhaps be a situation specific to unspooling > from disk while still intaking a largish volume of traffic?
Not sure. The system call overhead re synchronization (mutexes) should ot be higher, as unspooling still works in batches. Do you happen to remember how much the CPU utilization was for the input part? Rainer > > Brian > > On Thu, Nov 1, 2012 at 12:29 PM, Rainer Gerhards > <[email protected]> wrote: > > > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of Brian Knox > >> Sent: Wednesday, October 31, 2012 6:12 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or > redesign? > >> > >> I have two initial thoughts from experience concerning heavy use of > >> the queue system in high traffic (6+ billion log lines a day) > >> environments - specifically concerning DA queues. > >> > >> 1. The queues for the most part were very reliable > >> 2. The performance unspooling from a "fail" state (defined as "we > had > >> to spool to disk") was sub optimal. Using fast io subsystems also > >> seemed to have little impact on the unspool speed - it seemed stuck > at > >> around 50 to 60k messages a second in our environment, and seemed to > >> be more CPU bound than IO bound. > > > > This is pretty interesting to me. I'd always thought that the disk > queue system is heavily io bound, but uses very low CPU. I think I even > tested this, but not sure any longer. Performance was of no concern for > the initial design. > > > > Rainer > >> So, in our production we always tried to set the in memory portion > of > >> the queues very high - the disk backed portion was there in case of > a > >> disaster and we daily prayed we didn't end up having to fail over to > >> them. We lived in fear of finding ourselves in a situation where we > >> could not unspool fast enough to catch back up. > >> > >> The queue performance was a definite limiting factor for us. It was > >> probably are biggest pain point with rsyslog. > >> > >> I don't say this to trivialize the complexity of the problem - and > to > >> be clear, we rarely had -reliability- issues with the queues. > >> > >> Brian > >> > >> > >> On Wed, Oct 31, 2012 at 12:55 PM, Rainer Gerhards > >> <[email protected]> wrote: > >> > Hi all, > >> > > >> > There is the dangling issue that rsyslog has grown out of its > current > >> queue subsystem. I am currently considering a refactoring or a > complete > >> redesign. I initially wanted to write a large blog post with all > >> details and ideas, but have now opted to split this in a couple of > >> parts - both because I have problems to find time to do the "big > one" > >> at once; and also it probably is smarter to get feedback asap. > >> > > >> > So here is the initial part: > >> > > >> > http://blog.gerhards.net/2012/10/rsyslog-disk-queues-refactor- > or.html > >> > > >> > This will get anyone interested in the queue subsystem a broad > >> understanding of how it works - and why. Please share any concerns > you > >> have about the current system as well as wishes/suggestions on what > >> should improve. Deeply technical information is fine, actually > >> appreciated. > >> > > >> > I intend to let the discussion run and write the other parts of > the > >> blog series when "events warrant it" ;) Due to other projects, I can > >> probably not discuss 10 hours a day, but will try to be as active as > >> possible (which hopefully means "much"). The intent is to come up > with > >> a solution that will be good for the next five years to come... > >> > > >> > Thanks, > >> > Rainer > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> if you DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> if you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
