I hope there is a stats about metrics based on $programname, $severity, $fromhost-ip etc, extends the ruleset(impstats).
2015-10-07 16:19 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>: > -- > Regards, > Janmejay > > PS: Please blame the typos in this mail on my phone's uncivilized soft > keyboard sporting it's not-so-smart-assist technology. > > On Oct 7, 2015 3:26 AM, "David Lang" <da...@lang.hm> wrote: > > > > On Wed, 7 Oct 2015, singh.janmejay wrote: > > > >> -- > >> Regards, > >> Janmejay > >> > >> PS: Please blame the typos in this mail on my phone's uncivilized soft > >> keyboard sporting it's not-so-smart-assist technology. > >> > >> On Oct 6, 2015 10:32 PM, "David Lang" <da...@lang.hm> wrote: > >>> > >>> > >>> On Tue, 6 Oct 2015, singh.janmejay wrote: > >>> > >>>> It is possible to use global-variables (it'll require some > >>>> enhancements, table-support etc), but it'll be very inefficient > >>>> compared to this approach. For instance, choice of data-structure etc > >>>> allows making the solution a lot more efficient. > >>> > >>> > >>> > >>> As for the data structures, Rainer has been identifying inefficencies > in > >> > >> how json-c works and working to improve them > >>> > >>> > >> > >> That optimizes variable system. But it still is a general propose > variable > >> system. It can't and shouldn't understand relationship between > variables. > > > > > > what relationships are there between the different metrics? > > > > The fact that they are read in one shot, reported and reset. > > > > >>> > >>>> Here its possible to locklessly increment counters in most cases, so > >>>> its overhead is a lot lesser than global-variables. > >>> > >>> > >>> > >>> how can you manage counters in multiple threads without locks? > Especially > >> > >> when dealing with batches. > >>> > >>> > >> > >> Consider a trie based implementation. With bounded fanout-factor, it's > O(1) > >> wrt metric-names cardinality. It also has very little lock contention > >> involved. Usually operations work with read-locks, only when new metric > is > >> initialized it requires a write lock on patent node. If recycles are few > >> and far apart, lock contention would be negligible. > > > > > > if you have multiple threads that may need to update the same metric at > the same time, a tree doesn't eliminate the locking. > > > > The only situation involving a lock that is contended for, is when a metric > is to be initialized. Consider this trie: > > A -> B -> C > -> D > > Now for incrementing key ABC, no contention exists, because it involves > read-locks only. It just uses atomic-increment to bump the counter at node > C. Same for ABD. > > But ABE will require a write-lock at node B, because node E doesn't exist > yet. However key with not shared parent can again be initialized > concurrently. Init operation can be amortized over a large set of increment > operations making its cost negligible(this is the knob that reset interval > exposes). > > > The current json-c locking is being make intentionally over-broad right > now because it appears that some json-c code is not thread-safe and we > haven't identified it yet. Once that's tracked down and fixed (or json-c > replaced), updating one item should not require locking any more than that > item. > > > > > >>> > >>>> Recycle is precisely to allow this lockless mechanism to work. Its > >>>> basically saying, it'll track metric-names he has seen in last 1 hour. > >>>> If we kill tracking of it as soon as we don't see an increment > >>>> (between 2 reporting runs of impstats), it'll lead to unnecessary > >>>> churn when low-values are common or load is not uniform in time. > >>> > >>> > >>> > >>> that depends on the cost of initializing a metric vs the cost of > tracking > >> > >> the recycle mechanism. > >>> > >>> > >> > >> 0 value data-points can easily be filtered out. So they don't create any > >> processing overhead downstream. Cost of tracking for recycle is minimal > >> because it's a single counter bring tracked, when it reaches zero it's > >> reset to orig starting value and trie is killed after reporting > accumulated > >> stats. > > > > > > actually, filtering out 0 data-points can be a very bad thing. Far too > many monitoring tools produce stright-line graphs/estimates between > reported data-points, so it's very important to report 0 value data-points > > > > I agree. > > > > >>>> Implementing it on top of global-variables is not only has very high > >>>> performance-penalty(it'll be prohibitive for high-throughput > >>>> scenarios), it also exposes too much complexity to the user (where > >>>> user has to worry about reset etc). > >>>> > >>>> I don't plan to have a scheduler in this implementation. > >>>> GetAllStatsLines call will purge the tree instead of reset at that > >>>> interval. Its basically a balance between freeing-up memory occupied > >>>> by stale-metric-names vs. performance (lockless handling of > >>>> increment). So it will be governed by impstat schedule. May be I > >>>> should change name to better name (equivalent of > >>>> purge_known_keys_after_they_have_been_reported_N_times). > >>> > >>> > >>> > >>> if this is just adding additional metrics to the impstats output that > >> > >> eliminates the schedular/reset issue. > >>> > >>> > >>> I think we should have a metric configuration be fairly static, allow > >> > >> configuring custom metrics and add to them, but don't use data from the > >> message as part of the name of the metric, and continue reporting them > >> forever, even if they are 0 (so no need to 'recycle' names) > >> > >> Dynamic metrics are a real usecase for any shared system(utilisation > across > >> several users, several hosts, several clusters, several-subnets etc are > >> easily reportable with this). The only way to report utilisation in many > >> scenarios is to have dyn-metric names. The alternative is to pre-declare > >> all keys, but that to me is a more indirect solution. It's not as > >> flexible/adaptive. > >> > >> I think declarative static-key a useful feature on its own, for eg when > >> classifying reportable metric into buckets known in advance, but dyn-key > >> and configurable-static-key are not interchangeable. > > > > > > dynamic systems will have pathalogical failure condiditions. Consider > what happens when someone uses hostname in a dynafile template and then > some system starts spewing malformed logs that put garbage data in that > field. Creating hundreds or thousands of metric variables is much worse. > > The max-cardinality optional field in dyn-metric-namespace declaration was > exactly to prevent this kind of unbounded growth. We can choose a sensible > default. > > > > > I agree that pre-declared keys are less flexible, but they are also going > to be far safer and easier to deal with. > > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
