ACK! Please go, let's set aside my probably too strong concerns. I am perfectly happy when you say it's a small change and glad you do it!
Rainer Sent from phone, thus brief. Am 25.03.2016 06:21 schrieb "David Lang" <[email protected]>: > As I understand Rainer, go . > > David Lang > > On Fri, 25 Mar 2016, singh.janmejay wrote: > > Makes sense. So final call on foreach? Go or no-go? >> On Mar 25, 2016 2:38 AM, "David Lang" <[email protected]> wrote: >> >> Just a note, the dynastats output line needs this as well as the bucket >>> line. >>> >>> This isn't remote-input, but it needs the same ability to iterate over >>> things. >>> >>> example: >>> >>> >>> >>> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe ! >>> >> r_ > >> p! >> >>> >>> >>> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0} >>> >>> David Lang >>> >>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>> >>> Date: Wed, 23 Mar 2016 09:10:47 +0530 >>> >>>> From: singh.janmejay <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Subject: Re: [rsyslog] dynastats JSON output need work >>>> >>>> Agree, if most of us like it, I can implement foreach for objects. As of >>>> now I use omprog to call a script which uses jq and awk to make it >>>> opentsdb >>>> and redis friendly. I could have used omredis if foreach worked for >>>> objects. >>>> >>>> Let us conclude. >>>> >>>> @Rainer any thoughts on foreach for object? >>>> On Mar 23, 2016 1:56 AM, "David Lang" <[email protected]> wrote: >>>> >>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>> >>>>> >>>>> Makes sense. >>>>> >>>>> >>>>>> ..."counters":["a":80,"b":67] won't work, I think you meant >>>>>> ..."counters": [{"a" : 80}, {"b": 67}]. >>>>>> >>>>>> >>>>>> ahh, showing my lack of knowledge of JSON :-) talking things through: >>>>> >>>>> so our choices are: >>>>> >>>>> "counters": [{"a":80},{"b":67}] >>>>> vs >>>>> "counters": {"a":80,"b":67} >>>>> >>>>> running these through jq gives >>>>> >>>>> { >>>>> "counters": [ >>>>> { >>>>> "a": 80 >>>>> }, >>>>> { >>>>> "b": 67 >>>>> } >>>>> ] >>>>> } >>>>> >>>>> vs >>>>> >>>>> { >>>>> "counters": { >>>>> "a": 80, >>>>> "b": 67 >>>>> } >>>>> } >>>>> >>>>> or to reference a >>>>> .counters[0].a >>>>> vs >>>>> .counters.a >>>>> >>>>> The latter seems to clearly be the cleaner and more logical one to me. >>>>> >>>>> Does this seem sane? >>>>> >>>>> But to use it, we need a variation of foreach that gives the object >>>>> name >>>>> as well as the object contents. >>>>> >>>>> David Lang >>>>> >>>>> So it boils down to foreach handling object or not. >>>>> >>>>> >>>>>> Thoughts? >>>>>> >>>>>> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <[email protected]> wrote: >>>>>> >>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>>>> >>>>>>> >>>>>>> What about wrapDynamicObjects="on|off"? That is required regardless, >>>>>>> >>>>>>> right? (if we want to preserve backward compatibility). Unless we >>>>>>>> choose to change it anyway (which im fine with). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> I think the fact that data from the log could overwrite name or >>>>>>> origin >>>>>>> is a >>>>>>> fatal flaw and the existing format should not be allowed. So I think >>>>>>> we >>>>>>> should break backwards compatibility here (if it was more >>>>>>> established, >>>>>>> I'd >>>>>>> be much more reluctant to do so, but at this point, I think it's only >>>>>>> early >>>>>>> adopters who are using it) >>>>>>> >>>>>>> So if we are always going to push things down one level, the only >>>>>>> question >>>>>>> is the format >>>>>>> >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}} >>>>>>> >>>>>>> vs >>>>>>> >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]} >>>>>>> >>>>>>> or the horrible >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]} >>>>>>> >>>>>>> I really want to avoid this third one. I'd say it's probably better >>>>>>> to >>>>>>> have >>>>>>> to use an external script to process things than make the third one a >>>>>>> standard :-) >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>> @Rainer: what do you think about foreach handling object? >>>>>>> >>>>>>> >>>>>>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>>>>>>> >>>>>>>>> Foreach can only work with arrays as of now. It can't work with >>>>>>>>> >>>>>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is >>>>>>>>>> the only format that will work as of now. >>>>>>>>>> >>>>>>>>>> We can enhance foreach to work with objects. >>>>>>>>>> >>>>>>>>>> I can make a flag available at dyn-stats bucket level, which can >>>>>>>>>> control serialization format, but that would really be a hack. >>>>>>>>>> >>>>>>>>>> From single-responsibility pattern pov, impstats should be the >>>>>>>>>> only >>>>>>>>>> component that decides how to layout data for user to see. >>>>>>>>>> >>>>>>>>>> How about this: >>>>>>>>>> >>>>>>>>>> impstats(format="json" wrapDynamicObjects="on"...)? >>>>>>>>>> >>>>>>>>>> It defaults to off, which keeps backward compatibility. >>>>>>>>>> >>>>>>>>>> So what do you guys think about: >>>>>>>>>> - wrapDynamicObjects="on|off" >>>>>>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...} >>>>>>>>>> (foreach will handle the former out of the box, but later is >>>>>>>>>> concise, >>>>>>>>>> readable and light-weight in addition to being more json-y. >>>>>>>>>> - enhancing foreach to work with {a: 10, b: 20...} >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> If we can enhance foreach to work with the concise format, I would >>>>>>>>> rather >>>>>>>>> wait for it instead of introducing the wrapping version. >>>>>>>>> >>>>>>>>> I'm thinking that foreach walks through arrays, rather than mixing >>>>>>>>> concepts, >>>>>>>>> a foreachobject that gives us a name and contents for a {} list of >>>>>>>>> objects >>>>>>>>> may be the right thing to do? >>>>>>>>> >>>>>>>>> foreach just returns a single object while foreachobject needs to >>>>>>>>> return >>>>>>>>> the >>>>>>>>> object and name. >>>>>>>>> >>>>>>>>> although, if we ever get the ability to address arrays directly, >>>>>>>>> being >>>>>>>>> able >>>>>>>>> to look at the array position would be the equivalent of the name. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote: >>>>>>>>>>> >>>>>>>>>>> How about this: we support a new flag in impstats which allows >>>>>>>>>>> >>>>>>>>>>> json-formatted stats-line to optionally use >>>>>>>>>>>> encapsulated/wrapped-layout? >>>>>>>>>>>> >>>>>>>>>>>> impstats(format="json" ...) generates >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} >>>>>>>>>>>> >>>>>>>>>>>> however, >>>>>>>>>>>> >>>>>>>>>>>> impstats(format="json/w" ...) generates {"header": >>>>>>>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" : >>>>>>>>>>>> {"z-scribe1r-b":80,"SWEB10":67}} >>>>>>>>>>>> >>>>>>>>>>>> This is relevant, the serilization format we use right now mixes >>>>>>>>>>>> pre-defined keys with counter-names and it can affect regular >>>>>>>>>>>> static >>>>>>>>>>>> counters too. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> with the existing pstats output, there is no ability for >>>>>>>>>>> user-defined >>>>>>>>>>> data >>>>>>>>>>> to become a tag name, so there is no potential for ambiguity. but >>>>>>>>>>> with >>>>>>>>>>> dynastats, this is not a possibility, and the format we use >>>>>>>>>>> should >>>>>>>>>>> prevent >>>>>>>>>>> problems. >>>>>>>>>>> >>>>>>>>>>> Also, just from a conceptual point of view, why should the bucket >>>>>>>>>>> contents >>>>>>>>>>> be at the same level as the bucket name? >>>>>>>>>>> >>>>>>>>>>> other than backwards compatibility, what advantage is there of >>>>>>>>>>> the >>>>>>>>>>> current >>>>>>>>>>> version in JSON? the documentation uses the plain text >>>>>>>>>>> equivalant, >>>>>>>>>>> which >>>>>>>>>>> is >>>>>>>>>>> perfectly legitimate because there is an order to the line, and >>>>>>>>>>> after >>>>>>>>>>> you >>>>>>>>>>> get past the name and origin, everything else on the line is >>>>>>>>>>> name-value >>>>>>>>>>> pairs of counters, again, no ambiguity. >>>>>>>>>>> >>>>>>>>>>> But with JSON, I don't believe that you can depend on tools >>>>>>>>>>> maintaining >>>>>>>>>>> (or >>>>>>>>>>> even identifying) the order of the elements, and if you have >>>>>>>>>>> multiple >>>>>>>>>>> elements with the same name, it's implementation dependent as to >>>>>>>>>>> which >>>>>>>>>>> one >>>>>>>>>>> will be seen. >>>>>>>>>>> >>>>>>>>>>> So purely from a correctness and defensive programming point of >>>>>>>>>>> view, I >>>>>>>>>>> think the current JSON serialization should be changed, with the >>>>>>>>>>> old >>>>>>>>>>> format >>>>>>>>>>> no longer being an option. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> As to the details of the new format >>>>>>>>>>> >>>>>>>>>>> What I'm wanting to do with the counters is something like >>>>>>>>>>> >>>>>>>>>>> if $!origin == "dynstats.bucket" then { >>>>>>>>>>> foreach $.tag $!counters { >>>>>>>>>>> /var/log/stats;format >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> to output one line per counter. >>>>>>>>>>> >>>>>>>>>>> I'm very flexible in how to do this, but I would much rather be >>>>>>>>>>> able >>>>>>>>>>> to >>>>>>>>>>> do >>>>>>>>>>> this inside rsyslog than have to serialize things to an external >>>>>>>>>>> script, >>>>>>>>>>> have it parse the json and process it. >>>>>>>>>>> >>>>>>>>>>> my initial thinking was just do >>>>>>>>>>> >>>>>>>>>>> counters: [ "z-scribe1r-b":80,"SWEB10":67 ] >>>>>>>>>>> >>>>>>>>>>> but as I'm typing this, I realize that doesn't work as I don't >>>>>>>>>>> have a >>>>>>>>>>> way >>>>>>>>>>> to >>>>>>>>>>> break $.tag down to reference the name and the value. >>>>>>>>>>> >>>>>>>>>>> I'd hate to have to do something like >>>>>>>>>>> >>>>>>>>>>> counters: [{"name":"z-scribe1r-b","value":80 >>>>>>>>>>> },{"name":"SWEB10","value":67}] >>>>>>>>>>> >>>>>>>>>>> this mirrors the misuse of XML that gives it such a horrible >>>>>>>>>>> reputation. >>>>>>>>>>> But >>>>>>>>>>> unless we introduce some new function to rsyslog to break things >>>>>>>>>>> down, >>>>>>>>>>> I >>>>>>>>>>> don't see a better way. If we do need to do something like this, >>>>>>>>>>> I >>>>>>>>>>> sure >>>>>>>>>>> would not want to make it the default JSON, which would result in >>>>>>>>>>> two >>>>>>>>>>> different formats. I hate the idea of starting to have different >>>>>>>>>>> formats >>>>>>>>>>> because of subtypes of data (what is someone wants the cee >>>>>>>>>>> version >>>>>>>>>>> of >>>>>>>>>>> this >>>>>>>>>>> for example, you start to have orthoginal format decisions) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> David Lang >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>>> myriad >>>>>>>>>>> of >>>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>>> you >>>>>>>>>>> DON'T >>>>>>>>>>> LIKE THAT. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> myriad >>>>>>>>> of >>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>>>>> DON'T >>>>>>>>> LIKE THAT. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> >>>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>>> DON'T >>>>>>> LIKE THAT. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>>> >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
