An update a) and b) are more or less completed. I now shifted my approach a bit. I think I need to concentrate on the output plugin interface *before* I look at queue termination. The reason is that there is a very intimate relationship between the two. The rsyslog interface API dictates part of the termination process, but proper termination also dictates some of the interface details.
So to design and implement that, I'll be going down the full stack and then up again. That means a) add basic batching support to queue object b) create new batching output module interface c) add transactional behavior to queue shutdown and error conditions I have now arrived at step b). Today, I will need to think hard about the overall picture, but also about the plugin interface (plus do a couple of not-so-fun things ;)). So if you have any suggestions/opinions on the interface, please let me know. I will also see that I can extend the testbench, so that we can check the more complex conditions (this task is far more complex than it sounds, a lot of timing conditions are involved). Please also note that the multi-dequeue git branch now should work rather well (except, of course, when terminating the queue). The tcp input's performance has also enhanced. It would be very useful if someone could look at the new engine's speedup. It should be considerable. I expect that most of the potential speedup for non-database outputs is already gained in this version. Further speedup for e.g. the file writer or tcp forwarding output should be really marginal compared to what we gained with this step. So it is vital to confirm if it is well enough and within our expectations. I have not yet done performance tests, because they take a lot of time and I prefer not to be "disturbed" by such longer activity while I still work on the overall picture (even though I implement in parallel, think of this as proof of concept tasks - from my POV, I am looking at a very high level at the code and dig down only occasionally - longer testing task interrupt that process). Thanks, Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Rainer Gerhards > Sent: Wednesday, April 22, 2009 5:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] multi-dequeue git branch > > Oh, and I forgot to mention: it dequeues in batches, but the output > plugin > interface is unchanged. So we do not yet have the ability to add > batching to > the outputs. That's a totally different story, and I'll look at > implementation when we have done basic verification that the queue > works (or > earlier, as needs require ;)). > > Rainer > > > -----Original Message----- > > From: [email protected] [mailto:rsyslog- > > [email protected]] On Behalf Of Rainer Gerhards > > Sent: Wednesday, April 22, 2009 5:40 PM > > To: rsyslog-users > > Subject: [rsyslog] multi-dequeue git branch > > > > Hi all, > > > > I spent today on reviewing code and implementing a design approach > that > > I > > worked on the past days. Good thinking seems to pay well, and so I > > ended up > > with a version that should have performance benefits over the > previous > > versions. Actually, I hope that the performance is much better, even > > though > > the version currently dequeues in batches of 8 (soon to be > > configurable). If > > someone would like to give it a try, please use the "multi-dequeue" > git > > branch. I will be actively working on it. > > > > The current version is a proof of concept. It does NOT work fully > > correct. > > Most importantly, queue termination is not properly handled, nor are > > error > > conditions. If you use this version, you will probably lose message > on > > HUP, > > shutdown and when something goes wrong. I have also not yet done any > > performance testing myself. > > > > My planned next steps are > > > > a) extend the test bench to ensure that DA-mode works properly > > b) add config statements > > c) work on either the termination conditions or a new output plugin > > interface > > > > Especially c) will require lots of additional doc work, so there will > > hopefully some additional reading for you. I would also like to > mention > > that > > feedback in the current phase is very valuable. Most importantly, > once > > I have > > finalized the new algorithms, it will probably very hard to convince > me > > to > > change anything for the next couple of month. So if you are > interested > > in > > performance and the output plugin interface, it would be useful to > keep > > an > > eye on the list and provide comments where appropriate ;) > > > > Thanks, > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
