On Fri, Oct 25, 2013 at 12:48 PM, David Lang <[email protected]> wrote: > splitting this off as I'm going on a bit of a tangent here > > On Fri, 25 Oct 2013, Rainer Gerhards wrote: > > Also, as a slight tangent, if it's possible to have mmpstats variables >>> available as global variables (or similar) then much of the manual >>> counting >>> may not be needed as the pstats counters could be used instead. >>> >>> >> Could be done, but (probably) requires some time. Also note that these >> counts are NOT accurate - so they are only good if a rough number is used. >> Keeping them accurate would very considerably slow down the engine. But it >> definitely makes sense to have them available, for other uses as well >> (think about switching from one output to another based on the number of >> messages inside the main queue). I'd still consider this as a new feature >> request. >> > > I agree it's a different feature request, I think access to rsyslog > internal data (pstats, batch size, and future similar things) would be a > useful capability to get at some point. I was bringing it up now so that we > could try to make sure the design doesn't make it impossible to do this. > > You mention that pstats counts are not completely accurate for performance > reasons. > > First off, I want to make sure I understand the problem. > > That's actually something that is being documented for quite a while (not bashing, just being happy that something *is* documented ;)):
http://www.rsyslog.com/rsyslog-statistic-counter/ Is it that there could be races between different threads, or is it that > since the report takes time to generate, not all the numbers are from the > same point in time, and the process of reporting and then zeroing them may > race with them being modified? > > all of that. There are some operations for which no atomics exist, and putting them all under mutex protection would severely hinder rsyslog. Counters are taken via different methods, and some are accurate, some are not. But this may change as implementation changes, and I don't intend to document that. > > I'm wondering if the right approach here is to emulate the Linux Kernel > per-cpu variables, have each thread maintain it's own copy of any counters > that it's updating, and then have a way for the master thread to signal the > other threads that it needs them to report their values (and optionally > reset them) > > This would let each thread maintain their counters without expensive > atomic operations (which should improve performance if care is paid to > alignment and cacheline separation) and make everything as accurate as it > can be without having a global 'stop everything' flag while the counters > are read/reset, and even that may not be that bad if it's infrequent enough > (especially since the hotpath of updates would now be faster and rsyslog is > dealing with a handful of threads, not hundreds or thousands) > I agree this could be done -- but is it really worth the considerable effort. Just think about our mile-long TODO list. AFAIK, pstats works extremely well for what they were designed for (provide an idea of what's going on inside rsyslog). Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
