I personally would argue that stats around the actual content of syslog
messages is outside of the domain that rsyslog should be responsible for.
impstats makes sense to me as it provides statistics around rsyslogs
operation itself. Once I start wanting stats and counters around message
content, I would rather delegate that to a different system entirely.
On Tue, Oct 6, 2015 at 12:04 PM, singh.janmejay <singh.janme...@gmail.com>
wrote:
> Rainer,
>
> I see this as something completely outside the scope of variables.
> Building stats collector over variables is possible, but then we are
> then talking about a general purpose language which allows building
> such complex things. This increases the scope of Rainerscript and with
> larger scope comes complexity. I feel this is in-line with the other
> Lua discussion where you emphasized that Rainerscript should not
> become a fully-general-purpose language?
>
> Eg. creating an atomic-increment function for variable requires that
> we educate users about what can and can't be done if atomic-increment
> function is used anywhere on a variable. What relationship they can
> expect it to have with other atomic-incrementing variables (which gets
> into memory model).
>
>
>
> On Tue, Oct 6, 2015 at 8:49 PM, Rainer Gerhards
> <rgerha...@hq.adiscon.com> wrote:
> > I can't fully dig into this, but I think we must *very carefully*
> > evaluate the overall design. Some time ago we introduced arrays for
> > the limited liblognorm use case, and it hurts us every now and then
> > when folks want to use arrays for other use cases. It may probably
> > make sense to re-think how the variable engine etc behaves before
> > adding more functionality. And make sure that everything works smooth
> > in all use cases. While anything else may take care for some use
> > cases, I fear we may get too fragmented. At least this is what I
> > learned in the past months discussions.
> >
> > Anyone else?
> >
> > Rainer
> >
> > 2015-10-06 17:10 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
> >> It is possible to use global-variables (it'll require some
> >> enhancements, table-support etc), but it'll be very inefficient
> >> compared to this approach. For instance, choice of data-structure etc
> >> allows making the solution a lot more efficient.
> >>
> >> Here its possible to locklessly increment counters in most cases, so
> >> its overhead is a lot lesser than global-variables.
> >>
> >> Recycle is precisely to allow this lockless mechanism to work. Its
> >> basically saying, it'll track metric-names he has seen in last 1 hour.
> >> If we kill tracking of it as soon as we don't see an increment
> >> (between 2 reporting runs of impstats), it'll lead to unnecessary
> >> churn when low-values are common or load is not uniform in time.
> >>
> >> Implementing it on top of global-variables is not only has very high
> >> performance-penalty(it'll be prohibitive for high-throughput
> >> scenarios), it also exposes too much complexity to the user (where
> >> user has to worry about reset etc).
> >>
> >> I don't plan to have a scheduler in this implementation.
> >> GetAllStatsLines call will purge the tree instead of reset at that
> >> interval. Its basically a balance between freeing-up memory occupied
> >> by stale-metric-names vs. performance (lockless handling of
> >> increment). So it will be governed by impstat schedule. May be I
> >> should change name to better name (equivalent of
> >> purge_known_keys_after_they_have_been_reported_N_times).
> >>
> >>
> >> On Tue, Oct 6, 2015 at 4:30 PM, David Lang <da...@lang.hm> wrote:
> >>> On Tue, 6 Oct 2015, singh.janmejay wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I am working on support for stats with dynamic-name. This comes handy
> >>>> in situations where metric-name is dependent upon value of a certain
> >>>> attribute of the message.
> >>>>
> >>>> Say, for a central log-aggregation service, its valuable to know what
> >>>> is inbound message-count distribution across application-clusters that
> >>>> send logs to it, or for a shared-server, its valuable to know what is
> >>>> the log-volume generation across users etc.
> >>>>
> >>>> Im thinking of using functions-like interface to support this. It may
> >>>> look similar to this:
> >>>>
> >>>> ====================
> >>>> dyn_stats("user_msg_count")
> >>>>
> >>>> ...
> >>>>
> >>>> ruleset(...) {
> >>>> ...
> >>>> dyn_inc("user_msg_count", $.user)
> >>>> ...
> >>>> }
> >>>> ====================
> >>>>
> >>>> dyn_stats signature looks like:
> >>>> dyn_stats(<name_space>, <resettable: default=true>, <max_cardinality:
> >>>> default=10k>, <recycle_metric_names_after: default=1hr>)
> >>>>
> >>>> dyn_inc signature looks like:
> >>>> dyn_inc(<name_space>, <metric_name>)
> >>>>
> >>>>
> >>>> Reporting would work similar to static-metric via impstats. Mapping:
> >>>> statsobj_s.name = name_space
> >>>> statsobj_s.origin = "dyn"
> >>>> ctr_s.name = "foo" (say $.user had value foo)
> >>>>
> >>>>
> >>>> Thoughts / suggestions?
> >>>
> >>>
> >>> how is this different/better than global variables? (although we may
> need to
> >>> implement soem functions, atomic inc/dec copy+clear) If you have pstats
> >>> output in json format, you can even piggyback on it's schedule to
> output the
> >>> data.
> >>>
> >>>
> >>> things like stats can very easily end up being expensive in terms of
> locking
> >>> (something global variables already have figured out), and it sounds
> like
> >>> you are proposing adding a scheduler of some sort to output the data.
> >>>
> >>> variables should not need to be 'recycled', either they contain data
> or they
> >>> don't. If they contain data, you need to keep the data until you do
> >>> something with it, if they don't, you don't have to track them.
> >>>
> >>>
> >>> I am actually doing this sort of thing external to rsyslog in SEC
> >>>
> >>> I have a template in rsyslog that contains hostname, fromhost-ip,
> >>> programname and I output it via improg to SEC. SEC accumulates
> counters and
> >>> has scheduled outputs to files.
> >>>
> >>> before I started using SEC for this, I used the same template to
> output to a
> >>> file and then for reports, used cut + sort + uniq -c to extract the
> data I
> >>> need. When the files only contain the significant data, this is
> actually not
> >>> bad to do, even at higher volumes.
> >>>
> >>> David Lang
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of
> >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> >>> LIKE THAT.
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Janmejay
> >> http://codehunk.wordpress.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
