We resolved our issue with this by spawning more main Q threads (8) and removed the Action ques.
CPU usage for rsyslogd on the 50mb/sec box is sitting steadily at 200% (was 100). We will try and tune the regex as well. On Sat, Jun 15, 2013 at 5:09 AM, Rainer Gerhards <rgerha...@hq.adiscon.com>wrote: > I have updated the blog post (and fixed the typo): > > http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html > > Rainer > > > On Sat, Jun 15, 2013 at 11:57 AM, Rainer Gerhards > <rgerha...@hq.adiscon.com>wrote: > > > > > > > > > On Sat, Jun 15, 2013 at 9:49 AM, David Lang <da...@lang.hm> wrote: > > > >> In this particular case, there was one action in the ruleset (a rather > >> ugly dynafile with a regex in the filename and two in the format > string). > >> Based on this writeup it sounds as if multiple threads wouldn't help > >> (either on the mail or on the action queue) > >> > >> assuming the same number of actions, I am not understanding why there > >> would be any difference between trying to use multiple threads on a main > >> queue or an action queue. > >> > >> > > because more work happens on the main queue --> filtering, obtaining > > batches and a couple of minor other things. > > > > Rainer > > > >> by the way, I think you mix up sync and async at one point in the post > >> > >> "The reason is that main queue workers process all filters for all rules > >> inside the rule set, as well as full action processing for all actions > that > >> are not run on an synchronous (action) queue" > >> > >> shouldn't this be asynchronous (action) queue instead? > >> > >> > >> David Lang > >> > >> On Sat, 15 Jun 2013, Rainer Gerhards wrote: > >> > >> I managed to do some quick writeup. It's far from perfect, but may be > >>> useful: > >>> > >>> http://blog.gerhards.net/2013/**06/rsyslog-performance-main-** > >>> and-action.html< > http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html> > >>> > >>> > >>> If you have suggestions/questions, we can always extend or add more. > >>> Family > >>> is now asking me to go off the computer ;) > >>> > >>> Rainer > >>> > >>> > >>> On Sat, Jun 15, 2013 at 10:00 AM, Rainer Gerhards > >>> <rgerha...@hq.adiscon.com>**wrote: > >>> > >>> The effect on action queues is limited. Main q workers are the > primary > >>>> tool. Explaination in blog later. > >>>> > >>>> Sent from phone, thus brief. > >>>> Am 15.06.2013 09:46 schrieb "David Lang" <da...@lang.hm>: > >>>> > >>>> after he setup action queues it was only doing one thread per action > >>>> > >>>>> queue, even after the maxthreads for the actionqueue was set to 64. > >>>>> > >>>>> David Lang > >>>>> > >>>>> On Sat, 15 Jun 2013, Rainer Gerhards wrote: > >>>>> > >>>>> Action or main q threads? > >>>>> > >>>>>> > >>>>>> Sent from phone, thus brief. > >>>>>> Am 15.06.2013 09:23 schrieb "David Lang" <da...@lang.hm>: > >>>>>> > >>>>>> He had set the max threads to 64 and was still only seeing one > thread > >>>>>> > >>>>>>> working. > >>>>>>> > >>>>>>> In a few of the later messages in the thread, his actionqueue > config > >>>>>>> parameters are up near the top of the message. > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>> On Sat, 15 Jun 2013, Rainer Gerhards wrote: > >>>>>>> > >>>>>>> Requires a longer reply, need to lookup some things myself first - > >>>>>>> that's > >>>>>>> > >>>>>>> why I didn't throw in comments. > >>>>>>>> > >>>>>>>> But one very important thing: by default a queue has max 1 worker. > >>>>>>>> This > >>>>>>>> needs to be increased if multiple ones are desired. I guess that > >>>>>>>> was > >>>>>>>> the > >>>>>>>> main point missing. If you create an async action queue, you also > >>>>>>>> get > >>>>>>>> one > >>>>>>>> more thread. Probably that's why it looks how it looks. I don't > >>>>>>>> know > >>>>>>>> the > >>>>>>>> parameters out of my head. > >>>>>>>> > >>>>>>>> Sent from phone, thus brief. > >>>>>>>> Am 15.06.2013 03:40 schrieb "David Lang" <da...@lang.hm>: > >>>>>>>> > >>>>>>>> clarifying some questions that have come up in discussions around > >>>>>>>> > >>>>>>>> rulesets, thread, and queues > >>>>>>>>> > >>>>>>>>> When you tie a ruleset to a specific input, does that still use > the > >>>>>>>>> same > >>>>>>>>> main queue? or will the logs from that input go to a different > main > >>>>>>>>> queue > >>>>>>>>> (so the workers working on that queue will have no locking > >>>>>>>>> contention > >>>>>>>>> with > >>>>>>>>> workers working on logs from a different input), or do they get > put > >>>>>>>>> on a > >>>>>>>>> single main queue. > >>>>>>>>> > >>>>>>>>> how do output threads and rulesets interact? > >>>>>>>>> > >>>>>>>>> from what was reported, it looks as if everything goes to one > >>>>>>>>> queue, > >>>>>>>>> and > >>>>>>>>> only if you set action queues per-ruleset will you end up with > >>>>>>>>> per-ruleset > >>>>>>>>> threads (in this case, actually per-actionqueue threads) > >>>>>>>>> > >>>>>>>>> Is omfile able to make use of multiple threads if you have a > single > >>>>>>>>> action > >>>>>>>>> (in this case, using dynafiles) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> note, the documentation for omfile there is a bad entry > >>>>>>>>> http://www.rsyslog.com/doc/********omfile.html< > http://www.rsyslog.com/doc/******omfile.html> > >>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.**html< > http://www.rsyslog.com/doc/****omfile.html> > >>>>>>>>> > > >>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.html< > http://rsyslog.com/doc/**omfile.html> > >>>>>>>>> <http://www.**rsyslog.com/doc/**omfile.html< > http://www.rsyslog.com/doc/**omfile.html> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.html< > http://rsyslog.com/doc/**omfile.html> > >>>>>>>>> <http://rsyslog.**com/doc/omfile.html< > http://rsyslog.com/doc/omfile.html> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> <http://www.**rsyslog.com/doc/**omfile.html< > http://rsyslog.com/doc/omfile.html> > >>>>>>>>> <http://www.**rsyslog.com/doc/omfile.html< > http://www.rsyslog.com/doc/omfile.html> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> * DynaFileCacheSize (not mandatory, default will be used) > >>>>>>>>> Defines a template to be used for the output. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I think this is the wrong description and it would be good to say > >>>>>>>>> what > >>>>>>>>> the > >>>>>>>>> default is. > >>>>>>>>> > >>>>>>>>> David Lang > >>>>>>>>> ______________________________********_________________ > >>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/********mailman/listinfo/rsyslog< > http://lists.adiscon.net/******mailman/listinfo/rsyslog> > >>>>>>>>> <http**://lists.adiscon.net/******mailman/listinfo/rsyslog< > http://lists.adiscon.net/****mailman/listinfo/rsyslog> > >>>>>>>>> > > >>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog< > http://lists.adiscon.net/**mailman/**listinfo/rsyslog> > >>>>>>>>> <htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog< > http://lists.adiscon.net/**mailman/listinfo/rsyslog> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog< > http://lists.adiscon.net/**mailman/**listinfo/rsyslog> > >>>>>>>>> <htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog< > http://lists.adiscon.net/mailman/**listinfo/rsyslog> > >>>>>>>>> > > >>>>>>>>> <htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/**listinfo/rsyslog> > >>>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> http://www.rsyslog.com/********professional-services/< > http://www.rsyslog.com/******professional-services/> > >>>>>>>>>> <http://**www.rsyslog.com/******professional-services/< > http://www.rsyslog.com/****professional-services/> > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>> <http://**www.rsyslog.com/******professional-services/< > http://www.rsyslog.com/****professional-services/> > >>>>>>>>> <http://**www.rsyslog.com/****professional-services/< > http://www.rsyslog.com/**professional-services/> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> <http://**www.rsyslog.com/****professional-**services/< > http://www.rsyslog.com/**professional-**services/> > >>>>>>>>> <http:**//www.rsyslog.com/**professional-**services/< > http://www.rsyslog.com/professional-**services/> > >>>>>>>>> > > >>>>>>>>> <http:**//www.rsyslog.com/****professional-services/< > http://www.rsyslog.com/**professional-services/> > >>>>>>>>> <http://**www.rsyslog.com/professional-**services/< > http://www.rsyslog.com/professional-services/> > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>>> > >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>>> myriad > >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>>>>>>> you > >>>>>>>>> DON'T LIKE THAT. > >>>>>>>>> > >>>>>>>>> ______________________________******_________________ > >>>>>>>>> > >>>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog< > http://lists.adiscon.net/****mailman/listinfo/rsyslog> > >>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog< > http://lists.adiscon.net/**mailman/listinfo/rsyslog> > >>>>>>>> > > >>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/**listinfo/rsyslog> > >>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>>>> > > >>>>>>>> > >>>>>>>>> > >>>>>>>>> http://www.rsyslog.com/******professional-services/< > http://www.rsyslog.com/****professional-services/> > >>>>>>>> <http://**www.rsyslog.com/****professional-services/< > http://www.rsyslog.com/**professional-services/> > >>>>>>>> > > >>>>>>>> <http://**www.rsyslog.com/**professional-**services/< > http://www.rsyslog.com/professional-**services/> > >>>>>>>> <http:**//www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>>>>>> > > >>>>>>>> > >>>>>>>>> > >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>> myriad > >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>>> you > >>>>>>>> DON'T LIKE THAT. > >>>>>>>> > >>>>>>>> ______________________________******_________________ > >>>>>>>> > >>>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog< > http://lists.adiscon.net/****mailman/listinfo/rsyslog> > >>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog< > http://lists.adiscon.net/**mailman/listinfo/rsyslog> > >>>>>>> > > >>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/**listinfo/rsyslog> > >>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>>> > > >>>>>>> > >>>>>>>> > >>>>>>>> http://www.rsyslog.com/******professional-services/< > http://www.rsyslog.com/****professional-services/> > >>>>>>> <http://**www.rsyslog.com/****professional-services/< > http://www.rsyslog.com/**professional-services/> > >>>>>>> > > >>>>>>> <http://**www.rsyslog.com/**professional-**services/< > http://www.rsyslog.com/professional-**services/> > >>>>>>> <http:**//www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>>>>> > > >>>>>>> > >>>>>>>> > >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>> myriad > >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>> you > >>>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>>> ______________________________****_________________ > >>>>>>> > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog< > http://lists.adiscon.net/**mailman/listinfo/rsyslog> > >>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>> > > >>>>>> http://www.rsyslog.com/****professional-services/< > http://www.rsyslog.com/**professional-services/> > >>>>>> <http://**www.rsyslog.com/professional-**services/< > http://www.rsyslog.com/professional-services/> > >>>>>> > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>> myriad > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>>> DON'T LIKE THAT. > >>>>>> > >>>>>> ______________________________****_________________ > >>>>>> > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog< > http://lists.adiscon.net/**mailman/listinfo/rsyslog> > >>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>> > > >>>>> http://www.rsyslog.com/****professional-services/< > http://www.rsyslog.com/**professional-services/> > >>>>> <http://**www.rsyslog.com/professional-**services/< > http://www.rsyslog.com/professional-services/> > >>>>> > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> myriad > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>>> > >>>> ______________________________**_________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> ______________________________**_________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Tim Ehlers _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
