Okay, thanks for the explanation. I suspect I'm mis-using rsyslog and that I will be better off piping messages out omprog to a custom rewriter app and back in via a UNIX domain socket.
My original log messages come either from the Linux kernel via printk_emit or in Lumberjack form via ul_syslog. Both of those systems let me add structured data as key/value pairs. The rsyslog mmjsonparse finds the key/value data (as does imkmsg) - but that data is itself a flattened representation of hierarchical data. Neither printk_emit nor ul_syslog allow anything other than key/value string pairs AFAIK. I was trying to un-flatten that data during the rsyslog flow prior to putting it into mongodb. I very much understand the value in not creating a full-blown programming language for the rsyslog config files. I would suggest that the use case I'm describing will likely become more common over time. It might be possible to extend mmjsonparse by adding a pair of new parameters. One would be a boolean defaulting to "false" that indicates whether mmjsonparse should use the extended semantics. The second would be an optional seperator specifier defaulting to "!". The extended processing would expand the keys into the same JSON hierarchy that we get when we define $!foo!bar in rsyslog.conf. That would be powerful and require nothing more of the language. Effectively the daemon I intend to hang off of omprog will do this expansion and then write it back to rsyslog. Does that strike anybody as wrong-headed? I will ensure that there are no loops by gating off the message processing for the altered stream to do nothing but write it out via ommongodb. Thanks David Boles ________________________________________ From: [email protected] [[email protected]] on behalf of singh.janmejay [[email protected]] Sent: Friday, May 22, 2015 1:34 PM To: rsyslog-users Subject: Re: [rsyslog] Unable to use foreach Ok, we'll have to get into a little bit of code here, I know no better way to explain why it doesn't work for you, my apologies. This is the form that the array that you built is loaded to: https://github.com/rsyslog/rsyslog/blob/master/grammar/grammar.y#L215 We actually convert cnfarray to first element of that array for expr-evaluation: https://github.com/rsyslog/rsyslog/blob/master/grammar/rainerscript.c#L2314 Beyond this, all code assumes no cnfarray: https://github.com/rsyslog/rsyslog/blob/master/runtime/msg.c#L4558 So cnfarray was actually meant for a different purpose, I haven't yet dug into what that purpose is (@Rainer/David help?). But its definitely not equal to json-array, which is why your loop doesn't work. So in order to loop, you need a json array, which generally comes from mmjsonparse or mmnormalize generated object-graph. Take a look at the test in the PR I linked-to in this thread. That actually loops over an array and decides if it should stop processing a message depending on presence of a certain value in an array. That should serve as a working example of how to use foreach, except you'll probably have something useful in place of stop. On Fri, May 22, 2015 at 11:12 PM, David Lang <[email protected]> wrote: > On Fri, 22 May 2015, David Boles (dboles) wrote: > >> Yes, it is a string. Taking your clarification that foreach operates on >> arrays, I've tried the config below. When I push a log message from an >> application I observe: >> >> - The action with the msg1 template occurs >> - The action with the msg2 template does not occur >> - The action with the msg3 template occurs >> >> What should be changed below to cause the loop to be traversed? >> >> Thanks, >> >> David Boles >> >> -------- >> module(load="imuxsock") # local system logging support >> module(load="imkmsg") # structured kernel logging support >> >> module(load="mmjsonparse") >> >> template(name="msg1" type="list") { constant(value="pre-foreach record\n") >> } >> template(name="msg2" type="list") { constant(value=" foreach record\n") >> } >> template(name="msg3" type="list") { constant(value="post-foreach >> record\n") } >> >> if ($fromhost-ip == '127.0.0.1' and $syslogfacility-text != 'kern') then { >> action(type="mmjsonparse") >> if $parsesuccess == "OK" then { >> >> action(type="omfile" template="msg1" >> file="/var/log/db_local_user_structured.log") >> >> set $.myarray = [ "0", "1", "2", "3", "4", "5" ]; >> set $.index = 0; > > > there is no need for $.index = 0; it's not a counter, it just gets assigned > to each of the contents of myarray > >> foreach ($.index in $.myarray) do { >> action(type="omfile" template="msg2" >> file="/var/log/db_local_user_structured.log") >> } > > > this action should happen 6 times as I understand it. > >> action(type="omfile" template="msg3" >> file="/var/log/db_local_user_structured.log") >> } >> } >> -------- > > > having multiple action writing to the same file can result in some > interesting issues (at least conceptually, I'll let Rainer speak up if they > have been solved in practice) > > what I would do is > > $template custom,"%$.custom%" > > ruleset(name="db_local_user_structured"){ > action(type="omfile" template="custom" > file="/var/log/db_local_user_structured.log") > } > > and then replace each of the actions in your example with > > set $.custom = "message"; > call custom > > David Lang > > >> >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of singh.janmejay >> Sent: Thursday, May 21, 2015 10:43 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Unable to use foreach >> >> I think $!mse.element_indices is a string, right? >> >> If its a string, you'll need to parse it to make it an array before you >> can loop on it. Foreach works only with arrays. It can be array or anything >> (string, object, numbers whatever), but it has to be an array. >> >> You can use tokenized field-type to parse it and since you are already >> using mmjsonparse it shouldn't be a problem. >> >> On Thu, May 21, 2015 at 8:47 PM, David Boles (dboles) <[email protected]> >> wrote: >>> >>> Hi, >>> >>> I am using rsyslog (v8.9) to process structured log data from umberlog >>> and Linux's printk_emit. In the log message is a field >>> "$!mse.element_indices" that can have values such as "0", "0 1", "0 1 2", >>> and so on. I would like to iterate over the delimited elements of that value >>> and had supposed that foreach would do something like that. >>> >>> With the config below I comment/uncomment the foreach loop. When the >>> foreach loop is commented out rsyslog creates entries in both mongodb and >>> the file. When the foreach loop is uncommented, rsyslog produces nothing in >>> either destination. >>> >>> Why does this use of foreach fail? What should I be doing to iterate? >>> >>> Thanks, >>> >>> David Boles >>> >>> ---------------------------------------------------------------------- >>> -------- >>> >>> module(load="imuxsock") >>> module(load="imkmsg") >>> module(load="imtcp") >>> input(type="imtcp" port="10514") >>> module(load="mmjsonparse") >>> module(load="ommongodb") >>> >>> kern.* /var/log/db_kernel.log >>> *.* /var/log/db_full.log >>> >>> template(name="mse-structured-info" type="subtree" subtree="$!") >>> >>> template(name="mse-all-info" type="list") { >>> property(name="jsonmesg" outname="msg") } >>> >>> if ($fromhost-ip == '127.0.0.1' and $syslogfacility-text != 'kern') then >>> { >>> action(type="mmjsonparse") >>> if $parsesuccess == "OK" then { >>> set $!foo = $!mse.element_indices; >>> >>> set $!amph = "toad"; >>> # foreach ($.index in $!mse.element_indices) do { >>> # set $!amph = "turtle"; >>> # } >>> action(type="ommongodb" server="somemachine.somewhere.org" >>> db="logs" collection="syslog" >>> template="mse-structured-info") >>> >>> action(type="omfile" template="mse-all-info" >>> file="/var/log/db_local_user_structured.log") >>> } >>> } >>> >>> $WorkDirectory /var/spool/rsyslog >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> >> >> >> >> -- >> Regards, >> Janmejay >> http://codehunk.wordpress.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
