> > Yeah, but usually they should only happen if high reliability is > asked > > for. In the "normal" settings, they happen very infrequently. Thus I > > would be really interested to learn the source of the CPU > utilization. > > David, have you experienced something like this? > > I have probably found a main culprit for CPU load iff the .qi file is > synced very often: there is a sprintf() used to generate the filename, > and it is done each time the file is opened. I think this stemmed back > to the time when this function was only called at rsyslog termination. > I am moving this out to queue construction (it actually is a constant > string!). That should already relieve some of the CPU load. > > Still, this doesn't happen in default settings, so > > @Brian: could you try to reproduce this in a lab and help gather some > > info on what's going on? (I will also see if I can reproduce it > here).
I have looked further at the code. I think I see part of the inefficiency that happens in the regular case (with default parameters). The queue subsystem was originally written with a very generic scope, intended to handle things other than messages. However, we only process message objects via it (I need to check in detail if there are tiny exceptions, but I don't think so). This generic code has a high price, it uses various levels of indirection and does NOT use the optimized message creation paths e.g. it does a time system call for each message being read, even thought that time is immediately discarded. As past experience told us, that time call alone is very problematic. I also see other inefficiencies in regard to the encoding respective the way the in-memory object is restored. There is even a lot of room with the current encoding, not to mention the binary header David mentioned. I think a technically sound plan is to first make the queue-subsystem non-generic so that it knows what it deals with (messages!). With that extra information, I can than make the reader (hopefully much) more efficient. I think all other performance enhancements need to go after that. Moving the code to be msg-specific (non-generic) is required in any case as all other optimizations discussed so far rely on the fact that msg objects are being processed. Comments welcome; even more so folks who'd like to test the upcoming new stuff ;) Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
