> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Thursday, November 01, 2012 6:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or redesign? > > On Thu, 1 Nov 2012, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of Brian Knox > >> Sent: Wednesday, October 31, 2012 6:12 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or > redesign? > >> > >> I have two initial thoughts from experience concerning heavy use of > >> the queue system in high traffic (6+ billion log lines a day) > >> environments - specifically concerning DA queues. > >> > >> 1. The queues for the most part were very reliable > >> 2. The performance unspooling from a "fail" state (defined as "we > had > >> to spool to disk") was sub optimal. Using fast io subsystems also > >> seemed to have little impact on the unspool speed - it seemed stuck > at > >> around 50 to 60k messages a second in our environment, and seemed to > >> be more CPU bound than IO bound. > > > > This is pretty interesting to me. I'd always thought that the disk > queue > > system is heavily io bound, but uses very low CPU. I think I even > tested > > this, but not sure any longer. Performance was of no concern for the > > initial design. > > unless you are doing lots of fsyncs for data reliability (or are low on > ram), all your activity is happening in memory. the syscalls to open > and > close files are all cpu time for example.
Yeah, but usually they should only happen if high reliability is asked for. In the "normal" settings, they happen very infrequently. Thus I would be really interested to learn the source of the CPU utilization. David, have you experienced something like this? @Brian: could you try to reproduce this in a lab and help gather some info on what's going on? (I will also see if I can reproduce it here). Rainer > David Lang > > > Rainer > >> So, in our production we always tried to set the in memory portion > of > >> the queues very high - the disk backed portion was there in case of > a > >> disaster and we daily prayed we didn't end up having to fail over to > >> them. We lived in fear of finding ourselves in a situation where we > >> could not unspool fast enough to catch back up. > >> > >> The queue performance was a definite limiting factor for us. It was > >> probably are biggest pain point with rsyslog. > >> > >> I don't say this to trivialize the complexity of the problem - and > to > >> be clear, we rarely had -reliability- issues with the queues. > >> > >> Brian > >> > >> > >> On Wed, Oct 31, 2012 at 12:55 PM, Rainer Gerhards > >> <[email protected]> wrote: > >>> Hi all, > >>> > >>> There is the dangling issue that rsyslog has grown out of its > current > >> queue subsystem. I am currently considering a refactoring or a > complete > >> redesign. I initially wanted to write a large blog post with all > >> details and ideas, but have now opted to split this in a couple of > >> parts - both because I have problems to find time to do the "big > one" > >> at once; and also it probably is smarter to get feedback asap. > >>> > >>> So here is the initial part: > >>> > >>> http://blog.gerhards.net/2012/10/rsyslog-disk-queues-refactor- > or.html > >>> > >>> This will get anyone interested in the queue subsystem a broad > >> understanding of how it works - and why. Please share any concerns > you > >> have about the current system as well as wishes/suggestions on what > >> should improve. Deeply technical information is fine, actually > >> appreciated. > >>> > >>> I intend to let the discussion run and write the other parts of the > >> blog series when "events warrant it" ;) Due to other projects, I can > >> probably not discuss 10 hours a day, but will try to be as active as > >> possible (which hopefully means "much"). The intent is to come up > with > >> a solution that will be good for the next five years to come... > >>> > >>> Thanks, > >>> Rainer > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> if you DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> if you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
