Re: [rsyslog] rsyslog queue subsystem - refactor or redesign?

Thu, 01 Nov 2012 10:51:59 -0700 

On Thu, 1 Nov 2012, Rainer Gerhards wrote:


-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Brian Knox
Sent: Wednesday, October 31, 2012 6:12 PM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog queue subsystem - refactor or redesign?

I have two initial thoughts from experience concerning heavy use of
the queue system in high traffic (6+ billion log lines a day)
environments - specifically concerning DA queues.

1. The queues for the most part were very reliable
2. The performance unspooling from a "fail" state (defined as "we had
to spool to disk") was sub optimal.  Using fast io subsystems also
seemed to have little impact on the unspool speed - it seemed stuck at
around 50 to 60k messages a second in our environment, and seemed to
be more CPU bound than IO bound.
This is pretty interesting to me. I'd always thought that the disk queue system is heavily io bound, but uses very low CPU. I think I even tested this, but not sure any longer. Performance was of no concern for the initial design.
unless you are doing lots of fsyncs for data reliability (or are low on ram), all your activity is happening in memory. the syscalls to open and close files are all cpu time for example. 

David Lang


Rainer

So, in our production we always tried to set the in memory portion of
the queues very high - the disk backed portion was there in case of a
disaster and we daily prayed we didn't end up having to fail over to
them.  We lived in fear of finding ourselves in a situation where we
could not unspool fast enough to catch back up.

The queue performance was a definite limiting factor for us.  It was
probably are biggest pain point with rsyslog.

I don't say this to trivialize the complexity of the problem - and to
be clear, we rarely had -reliability- issues with the queues.

Brian


On Wed, Oct 31, 2012 at 12:55 PM, Rainer Gerhards
<[email protected]> wrote:

Hi all,

There is the dangling issue that rsyslog has grown out of its current

queue subsystem. I am currently considering a refactoring or a complete
redesign. I initially wanted to write a large blog post with all
details and ideas, but have now opted to split this in a couple of
parts - both because I have problems to find time to do the "big one"
at once; and also it probably is smarter to get feedback asap.


So here is the initial part:

http://blog.gerhards.net/2012/10/rsyslog-disk-queues-refactor-or.html

This will get anyone interested in the queue subsystem a broad

understanding of how it works - and why. Please share any concerns you
have about the current system as well as wishes/suggestions on what
should improve. Deeply technical information is fine, actually
appreciated.


I intend to let the discussion run and write the other parts of the

blog series when "events warrant it" ;) Due to other projects, I can
probably not discuss 10 hours a day, but will try to be as active as
possible (which hopefully means "much"). The intent is to come up with
a solution that will be good for the next five years to come...


Thanks,
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to