Sorry, *.conf files are ignored in .gitignore so I missed a file in mmnormalize test.
Here is the patch, it applies over the previous test patch(it does not replace it). On Fri, Oct 31, 2014 at 4:14 PM, Rainer Gerhards <[email protected]> wrote: > Thanks a lot! Merged... > > Rainer > > 2014-10-31 10:11 GMT+01:00 singh.janmejay <[email protected]>: > > > A test for this. > > > > > > On Fri, Oct 31, 2014 at 7:55 AM, singh.janmejay < > [email protected]> > > wrote: > > > > > Added doc for variable-support. > > > > > > Also fixed the ambiguous title (mmjsonparse and mmnormalize are now > > > identified in the title, hence on the index page). > > > > > > On Thu, Oct 30, 2014 at 5:08 PM, Rainer Gerhards < > > [email protected] > > > > wrote: > > > > > >> Right in the mmnormalize doc is the right spot: > > >> > > >> > > >> > > > https://github.com/rsyslog/rsyslog-doc/blob/master/source/configuration/modules/mmnormalize.rst > > >> > > >> Rainer > > >> > > >> 2014-10-30 12:28 GMT+01:00 singh.janmejay <[email protected]>: > > >> > > >> > What is the right place to add doc for this? > > >> > > > >> > On Wed, Oct 29, 2014 at 7:05 PM, Rainer Gerhards < > > >> [email protected] > > >> > > > > >> > wrote: > > >> > > > >> > > These two are finally merged... > > >> > > > > >> > > Rainer > > >> > > > > >> > > 2014-10-21 15:58 GMT+02:00 singh.janmejay < > [email protected] > > >: > > >> > > > > >> > > > No problem. > > >> > > > > > >> > > > -- > > >> > > > Regards, > > >> > > > Janmejay > > >> > > > > > >> > > > PS: Please blame the typos in this mail on my phone's > uncivilized > > >> soft > > >> > > > keyboard sporting it's not-so-smart-assist technology. > > >> > > > > > >> > > > On Oct 21, 2014 4:03 PM, "Rainer Gerhards" < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > >> > > > > Thx for the patches. Just FYI: I am deep inside complex work, > > will > > >> > look > > >> > > > at > > >> > > > > merging later the week, if things go really bad early next > week. > > >> > > > > > > >> > > > > Rainer > > >> > > > > > > >> > > > > 2014-10-21 9:32 GMT+02:00 singh.janmejay < > > >> [email protected]>: > > >> > > > > > > >> > > > > > On Tue, Oct 14, 2014 at 10:06 AM, singh.janmejay < > > >> > > > > [email protected] > > >> > > > > > > > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Tue, Oct 14, 2014 at 9:23 AM, David Lang < > [email protected]> > > >> > wrote: > > >> > > > > > > > > >> > > > > > >> On Tue, 14 Oct 2014, singh.janmejay wrote: > > >> > > > > > >> > > >> > > > > > >> Agree. > > >> > > > > > >>> > > >> > > > > > >>> I was not thinking of it like that. Will make the > > necessary > > >> > > > changes. > > >> > > > > > >>> > > >> > > > > > >>> Another question, do you want me to keep the template > > >> support > > >> > and > > >> > > > add > > >> > > > > > >>> variable support? > > >> > > > > > >>> > > >> > > > > > >>> or should I replace it completely with variable > support(im > > >> > > thinking > > >> > > > > > >>> replacing makes more sense)? > > >> > > > > > >>> > > >> > > > > > >> > > >> > > > > > >> I think replacing makes more sense. Since we have the > > >> ability to > > >> > > > set a > > >> > > > > > >> variable equal to the output of a template, not having a > > >> > template > > >> > > in > > >> > > > > the > > >> > > > > > >> action doesn't prevent you from doing anything, you just > > >> have to > > >> > > do > > >> > > > it > > >> > > > > > in > > >> > > > > > >> two steps instead of one. > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > > Sounds good, will rework it in the next patch. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > >> > > >> > > > > > >> David Lang > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> On Tue, Oct 14, 2014 at 8:53 AM, David Lang < > > [email protected]> > > >> > > wrote: > > >> > > > > > >>> > > >> > > > > > >>> On Tue, 14 Oct 2014, singh.janmejay wrote: > > >> > > > > > >>>> > > >> > > > > > >>>> Yep, sounds good. > > >> > > > > > >>>> > > >> > > > > > >>>>> > > >> > > > > > >>>>> The only concern I can think of: its different from > how > > >> other > > >> > > > > actions > > >> > > > > > >>>>> work. > > >> > > > > > >>>>> > > >> > > > > > >>>>> But that shouldn't be a problem if we make other > actions > > >> > accept > > >> > > > > > values > > >> > > > > > >>>>> too(and possibly deprecate usage of template in favour > > of > > >> > > > variable > > >> > > > > + > > >> > > > > > >>>>> exec_template). Something to think about, I guess. > > >> > > > > > >>>>> > > >> > > > > > >>>>> I'll make the change and send it back for review. > > >> > > > > > >>>>> > > >> > > > > > >>>>> > > >> > > > > > >>>> it's the difference between input and output. > > >> > > > > > >>>> > > >> > > > > > >>>> For other actions, the template is formatting the > output > > of > > >> > the > > >> > > > > > module, > > >> > > > > > >>>> and you almost never are going to be sending just a > > single > > >> > > > variable > > >> > > > > as > > >> > > > > > >>>> the > > >> > > > > > >>>> output (I can't think of any case, other than possibly > > >> rawmsg) > > >> > > > > > >>>> > > >> > > > > > >>>> In this case we are defining the input to the message > > >> > > modification > > >> > > > > > >>>> module, > > >> > > > > > >>>> msg, rawmsg, or an already defined variable. > > >> > > > > > >>>> > > >> > > > > > >>>> viewing it this way, I think it makes perfect sense > that > > >> it's > > >> > > not > > >> > > > a > > >> > > > > > >>>> template, but let's see what others think. > > >> > > > > > >>>> > > >> > > > > > >>>> David Lang > > >> > > > > > >>>> > > >> > > > > > >>>> > > >> > > > > > >>>> > > >> > > > > > >>>> On Tue, Oct 14, 2014 at 7:59 AM, David Lang < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > >>>>> > > >> > > > > > >>>>> As an efficiency thing, I would suggest having it > get a > > >> > > variable > > >> > > > > > >>>>> instead > > >> > > > > > >>>>> > > >> > > > > > >>>>>> of a templage. You can assign a template to a > variable > > if > > >> > you > > >> > > > need > > >> > > > > > to, > > >> > > > > > >>>>>> but > > >> > > > > > >>>>>> if you don't (because a prior step already parsed out > > the > > >> > data > > >> > > > to > > >> > > > > a > > >> > > > > > >>>>>> variable) you skip the time needed to go through the > > >> > template > > >> > > > > > engine. > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> I expect that most of the time you are going to have > > the > > >> > part > > >> > > > you > > >> > > > > > >>>>>> want to > > >> > > > > > >>>>>> normalize as the contents of a variable being passed > in > > >> via > > >> > > JSON > > >> > > > > or > > >> > > > > > a > > >> > > > > > >>>>>> structured data field, not as something that you > would > > >> need > > >> > > the > > >> > > > > full > > >> > > > > > >>>>>> power > > >> > > > > > >>>>>> and complexity of the template string generator to > get > > >> at. > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> The template string handler is slower than you would > > >> think. > > >> > > When > > >> > > > > we > > >> > > > > > >>>>>> added > > >> > > > > > >>>>>> the ability to have string generation modules in C > > >> instead > > >> > of > > >> > > > > using > > >> > > > > > >>>>>> the > > >> > > > > > >>>>>> template engine, it made a noticable difference in > the > > >> total > > >> > > > > > >>>>>> throughput > > >> > > > > > >>>>>> of > > >> > > > > > >>>>>> rsyslog, and if we are just going to be starting > from a > > >> > string > > >> > > > to > > >> > > > > > >>>>>> begin > > >> > > > > > >>>>>> with, running it through the template engine is pure > > >> > overhead. > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> Thanks for doing this sort of thing. I was wanting to > > get > > >> > > > > > mmnormalize > > >> > > > > > >>>>>> and > > >> > > > > > >>>>>> mmjsonparse to work on arbitrary strings instead of > > just > > >> > $msg, > > >> > > > but > > >> > > > > > >>>>>> hadn't > > >> > > > > > >>>>>> had time to deal with it. > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> David Lang > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> On Tue, 14 Oct 2014, singh.janmejay wrote: > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> Date: Tue, 14 Oct 2014 01:03:44 +0530 > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> From: singh.janmejay <[email protected]> > > >> > > > > > >>>>>>> Reply-To: rsyslog-users <[email protected]> > > >> > > > > > >>>>>>> To: [email protected] > > >> > > > > > >>>>>>> Subject: [rsyslog] Patch to allow choice of > templates > > in > > >> > > > > > mmnormalize > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> Hi, > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> As of now mmnormalize supports only raw-msg or msg > via > > >> the > > >> > > > > boolean > > >> > > > > > >>>>>>> flag > > >> > > > > > >>>>>>> 'useRawMsg'. This patch allows using > > >> > template="template_name" > > >> > > > > with > > >> > > > > > >>>>>>> mmnormalize just like a few other actions(such as > > omfwd, > > >> > > > > > >>>>>>> omelasticsearch > > >> > > > > > >>>>>>> etc) do. > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> It expects a param called 'template' following the > > >> > convention > > >> > > > in > > >> > > > > > >>>>>>> other > > >> > > > > > >>>>>>> action-types. And when both useRawMsg and template > > >> params > > >> > are > > >> > > > > set, > > >> > > > > > it > > >> > > > > > >>>>>>> prints a warning that template will be ignored and > > uses > > >> > > > > > raw-message > > >> > > > > > >>>>>>> instead. > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> The patch is called 0003... because its the 3rd > patch > > >> in my > > >> > > > > > >>>>>>> local-repo. > > >> > > > > > >>>>>>> But > > >> > > > > > >>>>>>> since it doesn't have any files common with previous > > >> > patches, > > >> > > > it > > >> > > > > > >>>>>>> should > > >> > > > > > >>>>>>> apply just fine on any recent commit. > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> Please let me know if the patch looks good for > > merging, > > >> > I'll > > >> > > > > update > > >> > > > > > >>>>>>> the > > >> > > > > > >>>>>>> documentation. > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> _______________________________________________ > > >> > > > > > >>>>>>> > > >> > > > > > >>>>>>> rsyslog mailing list > > >> > > > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > > > > >>>>>> http://www.rsyslog.com/professional-services/ > > >> > > > > > >>>>>> What's up with rsyslog? Follow > > >> > https://twitter.com/rgerhards > > >> > > > > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are > > >> ARCHIVED > > >> > > by > > >> > > > a > > >> > > > > > >>>>>> myriad > > >> > > > > > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and > DO > > >> NOT > > >> > > POST > > >> > > > if > > >> > > > > > you > > >> > > > > > >>>>>> DON'T LIKE THAT. > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> > > >> > > > > > >>>>> > > >> > > > > > >>>>> > > >> > > > > > >>>>> _______________________________________________ > > >> > > > > > >>>>> > > >> > > > > > >>>> rsyslog mailing list > > >> > > > > > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > > > > >>>> http://www.rsyslog.com/professional-services/ > > >> > > > > > >>>> What's up with rsyslog? Follow > > >> https://twitter.com/rgerhards > > >> > > > > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are > > >> ARCHIVED > > >> > by > > >> > > a > > >> > > > > > myriad > > >> > > > > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO > > NOT > > >> > POST > > >> > > if > > >> > > > > you > > >> > > > > > >>>> DON'T LIKE THAT. > > >> > > > > > >>>> > > >> > > > > > >>>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> _______________________________________________ > > >> > > > > > >> rsyslog mailing list > > >> > > > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > > > > >> http://www.rsyslog.com/professional-services/ > > >> > > > > > >> What's up with rsyslog? Follow > > https://twitter.com/rgerhards > > >> > > > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are > > ARCHIVED > > >> by > > >> > a > > >> > > > > myriad > > >> > > > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT > > >> POST > > >> > if > > >> > > > you > > >> > > > > > >> DON'T LIKE THAT. > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > -- > > >> > > > > > > Regards, > > >> > > > > > > Janmejay > > >> > > > > > > http://codehunk.wordpress.com > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > Hi, > > >> > > > > > > > >> > > > > > Sorry for the loss the continuity. > > >> > > > > > > > >> > > > > > Here are the patches for mmnormalize variable support. > > >> > > > > > > > >> > > > > > I have left patch 0001-... as is (it is the same old patch > > which > > >> > > > enables > > >> > > > > > template support) > > >> > > > > > > > >> > > > > > The patch 0002-... follows that and replaces support for > > >> templates > > >> > > with > > >> > > > > > variable. > > >> > > > > > > > >> > > > > > My mmnormalize invocation now looks like this: > > >> > > > > > > > >> > > > > > action(type="mmnormalize" ruleBase="/tmp/foo.rulebase" > > >> > > > variable="$.foo") > > >> > > > > > > > >> > > > > > Variable can be any property or a local/global variable etc. > > It > > >> > uses > > >> > > > > > MsgGetProp, so it should support everything. > > >> > > > > > > > >> > > > > > -- > > >> > > > > > Regards, > > >> > > > > > Janmejay > > >> > > > > > http://codehunk.wordpress.com > > >> > > > > > > > >> > > > > > _______________________________________________ > > >> > > > > > rsyslog mailing list > > >> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > > > > http://www.rsyslog.com/professional-services/ > > >> > > > > > What's up with rsyslog? Follow > https://twitter.com/rgerhards > > >> > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > > by > > >> a > > >> > > > myriad > > >> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > POST > > >> if > > >> > > you > > >> > > > > > DON'T LIKE THAT. > > >> > > > > > > > >> > > > > _______________________________________________ > > >> > > > > rsyslog mailing list > > >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > > > http://www.rsyslog.com/professional-services/ > > >> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by > > a > > >> > > myriad > > >> > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > > if > > >> > you > > >> > > > > DON'T LIKE THAT. > > >> > > > > > > >> > > > _______________________________________________ > > >> > > > rsyslog mailing list > > >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > > http://www.rsyslog.com/professional-services/ > > >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > > >> > myriad > > >> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > > >> you > > >> > > > DON'T LIKE THAT. > > >> > > > > > >> > > _______________________________________________ > > >> > > rsyslog mailing list > > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > > http://www.rsyslog.com/professional-services/ > > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >> myriad > > >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > >> > > DON'T LIKE THAT. > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Regards, > > >> > Janmejay > > >> > http://codehunk.wordpress.com > > >> > _______________________________________________ > > >> > rsyslog mailing list > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > http://www.rsyslog.com/professional-services/ > > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >> > DON'T LIKE THAT. > > >> > > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > > > > > > > > > > > -- > > > Regards, > > > Janmejay > > > http://codehunk.wordpress.com > > > > > > > > > > > -- > > Regards, > > Janmejay > > http://codehunk.wordpress.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Regards, Janmejay http://codehunk.wordpress.com
From bb5e03bed1fc0a8db1782f410c4730f322316946 Mon Sep 17 00:00:00 2001 From: Janmejay Singh <[email protected]> Date: Fri, 31 Oct 2014 18:44:10 +0530 Subject: [PATCH] *.conf files are ignored in .gitignore, so this file was missed in the mmnormalize test patch --- tests/testsuites/mmnormalize_variable.conf | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 tests/testsuites/mmnormalize_variable.conf diff --git a/tests/testsuites/mmnormalize_variable.conf b/tests/testsuites/mmnormalize_variable.conf new file mode 100644 index 0000000..43c609e --- /dev/null +++ b/tests/testsuites/mmnormalize_variable.conf @@ -0,0 +1,15 @@ +$IncludeConfig diag-common.conf +template(name="outfmt" type="string" string="h:%$!hr% m:%$!min% s:%$!sec%\n") + +module(load="../plugins/mmnormalize/.libs/mmnormalize") +module(load="../plugins/imptcp/.libs/imptcp") +input(type="imptcp" port="13514") + +template(name="time_fragment" type="list") { + property(name="msg" regex.Expression="[0-9]{2}:[0-9]{2}:[0-9]{2} [A-Z]+" regex.Type="ERE" regex.Match="0") +} + +set $.time_frag = exec_template("time_fragment"); + +action(type="mmnormalize" rulebase="testsuites/mmnormalize_variable.rulebase" variable="$.time_frag") +action(type="omfile" file="./rsyslog.out.log" template="outfmt") -- 2.0.4
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
