> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Wednesday, January 26, 2011 10:39 AM > To: rsyslog-users > Subject: Re: [rsyslog] message manipulation in a parser > > the message parser is turning out to be extremely easy to use in this > mode. > > I was able to re-write the cisconames fixup into the aixforwardedfrom > fixup in about 2 hours, and the completed patch (including tieing it in > to > the build system) is only 288 lines.
I am glad it worked out to be easy -- this is actually how it was designed :) > > It may very well be worth taking a bit of time to look at the default > rfc3164 parser and see how much it could be simplified by pulling some > of > the odd corner cases out to fixup parsers like this. > > it's not worth doing for flaws that show up frequently (having the code > in > the rfx3164 parser is going to be faster than going in to a separate > parser, not to mention the config complexity), but some of the more > strange cases may be worth pulling out, not as much for performance > (although it will probably help a smidge there), but more so in terms > of > simplifying the code and documenting what generates strange messages. I think the legacy parser is a good compromise, and that's one reason why I did not tear it apart. It contains cases which frequently occur, at least I think so. Also the guesswork should not cost too much performance. > Also, I saw your tweet about imttcp and was wondering, with a small > number > of threads, was it enough faster that it could be useful for the second > tier of a multi-tier logging infrastructure, where you have your first > tier collecting from a lot of machines to a small number of machines, > and > then this first tier is sending the logs (at a fairly high volume per > connection) to the next tier up? this is the opposite of the case you > were testing (which as I understand it was lots of source machines, > each > with a fairly low log volume) Possibly, 5% to 10%, but with a very low number of connections. I left the code inside the tree. It is not bug free, but enough to try it out. Anyhow, this was just a re-verification of my concepts. I will now see that I change imtcp so that it can utilize a small number of worker threads. That should bring the same effect, but scale better. It requires some steps before I can actually do that, but I am progressing. Rainer > it may not be worth the overhead of maintaining a separate input > module, > but I thought I'd mention the possibility. > > David Lang > > > On Wed, 26 Jan 2011, Rainer Gerhards > wrote: > > > Excellent, thx. I'll see that I merge both into the main branch > today. Right > > now my main machine is running a performance benchmark, so I need to > do this > > later today. > > > > Rainer > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of [email protected] > >> Sent: Wednesday, January 26, 2011 10:06 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] message manipulation in a parser > >> > >> I tested the code in your branch and verified that it works. > >> > >> David Lang > >> > >> On Tue, 25 Jan 2011, Rainer Gerhards wrote: > >> > >>> David, > >>> > >>> I now integrated the patch. I think I will also move it into the > main > >> branch > >>> soon. > >>> > >>> Rainer > >>> > >>> On 01/17/2011 10:15 AM, [email protected] wrote: > >>>> On Mon, 17 Jan 2011, Rainer Gerhards wrote: > >>>> > >>>>> Date: Mon, 17 Jan 2011 09:53:24 +0100 > >>>>>> [email protected]] On Behalf Of [email protected] > >>>>>> > >>>>>> On Mon, 17 Jan 2011, Rainer Gerhards wrote: > >>>>>> > >>>>>>> I guess it makes sense when you send me the code and give the > >> line > >>>>>>> number where you have the issue. I can't promise, but I'll try > to > >>>>>>> give a good suggestion in the course of the day. > >>>>>>> > >>>>>>> One important thing you need to think about is that pMsg does > NOT > >>>>>>> necessarily use cstr objects to represent the entities. For > >> example > >>>>>>> (IIRC), raw message is kept in a different buffer, and you > cannot > >>>>>>> use cstr methods on that buffer. %msg% is actually not stored > at > >>>>>>> all, it is taken from rawmsg. There is a whole lot of this kind > >> of > >>>>>>> optimization done inside the message object. I guess this is > what > >> is > >>>>>>> hitting you. > >>>>>> > >>>>>> Ok, that makes sense. so should I just insert a \0 at the > >> appropriate > >>>>>> place and decrament the iLenRawMsg and iLenMSG values? > >>>>> > >>>>> Tob e 100% sure, I'd need to check more thoroughly, but I am sure > >> you > >>>>> are on > >>>>> the right path... But have a look at msg.c. Where the buffer is > >> located > >>>>> depends on the size of it (it is static inside the object up to a > >> certain > >>>>> size, and dyn alloced otherwise -- another optimization). The > safe > >> bet > >>>>> is to > >>>>> duplicate the message text and re-set it, but that is obviously > >> much > >>>>> slower > >>>>> (to gain speed, you need to break layers...). > >>>> > >>>> Ok, This is now working, I think I found all the variables to > >> change. > >>>> > >>>> the patch on top of v5-devel-david is > >>>> > >>>> diff --git a/plugins/pmcisconames/pmcisconames.c > >>>> b/plugins/pmcisconames/pmcisconames.c > >>>> index 28fe33d..7b76f65 100644 > >>>> --- a/plugins/pmcisconames/pmcisconames.c > >>>> +++ b/plugins/pmcisconames/pmcisconames.c > >>>> @@ -41,7 +41,7 @@ > >>>> #include "unicode-helper.h" > >>>> > >>>> MODULE_TYPE_PARSER > >>>> -PARSER_NAME("rsyslog.lastline") > >>>> +PARSER_NAME("rsyslog.cisconames") > >>>> > >>>> /* internal structures > >>>> */ > >>>> @@ -108,9 +108,15 @@ dbgprintf("not a cisco name mangled log!\n"); > >>>> ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE); > >>>> } > >>>> /* bump the message portion up by two characters to overwrite the > >> extra > >>>> : */ > >>>> - memmove(p2parse, p2parse + 2, lenMsg - 2); > >>>> + lenMsg -=2; > >>>> + memmove(p2parse, p2parse + 2, lenMsg); > >>>> + *(p2parse + lenMsg) = '\n'; > >>>> + *(p2parse + lenMsg + 1) = '\0'; > >>>> + pMsg->iLenRawMsg -=2; > >>>> + pMsg->iLenMSG -=2; > >>>> /* now, claim to abort so that something else can parse the now > >> modified > >>>> message */ > >>>> DBGPRINTF("pmcisconames detected a mangled Cisco log message > >> message\n"); > >>>> + dbgprintf("pmcisconames: new mesage: [%d]'%s'\n", lenMsg, > >> p2parse); > >>>> ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE); > >>>> > >>>> finalize_it: > >>>> @@ -143,7 +149,7 @@ CODEmodInit_QueryRegCFSLineHdlr > >>>> CHKiRet(objUse(parser, CORE_COMPONENT)); > >>>> CHKiRet(objUse(datetime, CORE_COMPONENT)); > >>>> > >>>> - dbgprintf("lastmsg parser init called, compiled with version > >> %s\n", > >>>> VERSION); > >>>> + dbgprintf("cisconames parser init called, compiled with version > >> %s\n", > >>>> VERSION); > >>>> bParseHOSTNAMEandTAG = glbl.GetParseHOSTNAMEandTAG(); /* cache > >> value, is > >>>> set only during rsyslogd option processing */ > >>>> > >>>> > >>>> > >>>> David Lang > >>>> > >>>> P.S. I think having the file named pmlastmsg but the intermal > module > >>>> name being lastline is confusing, they should be related. > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
