Rainer - I can try to make some space to build from git head tomorrow and test out the queue changes on one of our lab servers.
Brian On Mon, Nov 5, 2012 at 2:50 AM, Rainer Gerhards <[email protected]> wrote: >> I have looked further at the code. I think I see part of the >> inefficiency that happens in the regular case (with default >> parameters). The queue subsystem was originally written with a very >> generic scope, intended to handle things other than messages. However, >> we only process message objects via it (I need to check in detail if >> there are tiny exceptions, but I don't think so). This generic code has >> a high price, it uses various levels of indirection and does NOT use >> the optimized message creation paths e.g. it does a time system call >> for each message being read, even thought that time is immediately >> discarded. As past experience told us, that time call alone is very >> problematic. >> >> I also see other inefficiencies in regard to the encoding respective >> the way the in-memory object is restored. There is even a lot of room >> with the current encoding, not to mention the binary header David >> mentioned. >> >> I think a technically sound plan is to first make the queue-subsystem >> non-generic so that it knows what it deals with (messages!). With that >> extra information, I can than make the reader (hopefully much) more >> efficient. I think all other performance enhancements need to go after >> that. Moving the code to be msg-specific (non-generic) is required in >> any case as all other optimizations discussed so far rely on the fact >> that msg objects are being processed. > > The current git master head now contains a refactored version that implements > many of the things I mentioned above. On my development machine, I get around > a 10 to 15% speedup (but this number is always questionable, as it is > virtualized, no testing done on real hardware). > > I'll try today to check a little bit more on the refactoring approach, I > think there is at least one more spot that I can change with relatively > little effort (and hopefully big effect). > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
