the message parser is turning out to be extremely easy to use in this
mode.
I was able to re-write the cisconames fixup into the aixforwardedfrom
fixup in about 2 hours, and the completed patch (including tieing it in to
the build system) is only 288 lines.
It may very well be worth taking a bit of time to look at the default
rfc3164 parser and see how much it could be simplified by pulling some of
the odd corner cases out to fixup parsers like this.
it's not worth doing for flaws that show up frequently (having the code in
the rfx3164 parser is going to be faster than going in to a separate
parser, not to mention the config complexity), but some of the more
strange cases may be worth pulling out, not as much for performance
(although it will probably help a smidge there), but more so in terms of
simplifying the code and documenting what generates strange messages.
Also, I saw your tweet about imttcp and was wondering, with a small number
of threads, was it enough faster that it could be useful for the second
tier of a multi-tier logging infrastructure, where you have your first
tier collecting from a lot of machines to a small number of machines, and
then this first tier is sending the logs (at a fairly high volume per
connection) to the next tier up? this is the opposite of the case you
were testing (which as I understand it was lots of source machines, each
with a fairly low log volume)
it may not be worth the overhead of maintaining a separate input module,
but I thought I'd mention the possibility.
David Lang
On Wed, 26 Jan 2011, Rainer Gerhards
wrote:
Excellent, thx. I'll see that I merge both into the main branch today. Right
now my main machine is running a performance benchmark, so I need to do this
later today.
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of [email protected]
Sent: Wednesday, January 26, 2011 10:06 AM
To: rsyslog-users
Subject: Re: [rsyslog] message manipulation in a parser
I tested the code in your branch and verified that it works.
David Lang
On Tue, 25 Jan 2011, Rainer Gerhards wrote:
David,
I now integrated the patch. I think I will also move it into the main
branch
soon.
Rainer
On 01/17/2011 10:15 AM, [email protected] wrote:
On Mon, 17 Jan 2011, Rainer Gerhards wrote:
Date: Mon, 17 Jan 2011 09:53:24 +0100
[email protected]] On Behalf Of [email protected]
On Mon, 17 Jan 2011, Rainer Gerhards wrote:
I guess it makes sense when you send me the code and give the
line
number where you have the issue. I can't promise, but I'll try to
give a good suggestion in the course of the day.
One important thing you need to think about is that pMsg does NOT
necessarily use cstr objects to represent the entities. For
example
(IIRC), raw message is kept in a different buffer, and you cannot
use cstr methods on that buffer. %msg% is actually not stored at
all, it is taken from rawmsg. There is a whole lot of this kind
of
optimization done inside the message object. I guess this is what
is
hitting you.
Ok, that makes sense. so should I just insert a \0 at the
appropriate
place and decrament the iLenRawMsg and iLenMSG values?
Tob e 100% sure, I'd need to check more thoroughly, but I am sure
you
are on
the right path... But have a look at msg.c. Where the buffer is
located
depends on the size of it (it is static inside the object up to a
certain
size, and dyn alloced otherwise -- another optimization). The safe
bet
is to
duplicate the message text and re-set it, but that is obviously
much
slower
(to gain speed, you need to break layers...).
Ok, This is now working, I think I found all the variables to
change.
the patch on top of v5-devel-david is
diff --git a/plugins/pmcisconames/pmcisconames.c
b/plugins/pmcisconames/pmcisconames.c
index 28fe33d..7b76f65 100644
--- a/plugins/pmcisconames/pmcisconames.c
+++ b/plugins/pmcisconames/pmcisconames.c
@@ -41,7 +41,7 @@
#include "unicode-helper.h"
MODULE_TYPE_PARSER
-PARSER_NAME("rsyslog.lastline")
+PARSER_NAME("rsyslog.cisconames")
/* internal structures
*/
@@ -108,9 +108,15 @@ dbgprintf("not a cisco name mangled log!\n");
ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE);
}
/* bump the message portion up by two characters to overwrite the
extra
: */
- memmove(p2parse, p2parse + 2, lenMsg - 2);
+ lenMsg -=2;
+ memmove(p2parse, p2parse + 2, lenMsg);
+ *(p2parse + lenMsg) = '\n';
+ *(p2parse + lenMsg + 1) = '\0';
+ pMsg->iLenRawMsg -=2;
+ pMsg->iLenMSG -=2;
/* now, claim to abort so that something else can parse the now
modified
message */
DBGPRINTF("pmcisconames detected a mangled Cisco log message
message\n");
+ dbgprintf("pmcisconames: new mesage: [%d]'%s'\n", lenMsg,
p2parse);
ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE);
finalize_it:
@@ -143,7 +149,7 @@ CODEmodInit_QueryRegCFSLineHdlr
CHKiRet(objUse(parser, CORE_COMPONENT));
CHKiRet(objUse(datetime, CORE_COMPONENT));
- dbgprintf("lastmsg parser init called, compiled with version
%s\n",
VERSION);
+ dbgprintf("cisconames parser init called, compiled with version
%s\n",
VERSION);
bParseHOSTNAMEandTAG = glbl.GetParseHOSTNAMEandTAG(); /* cache
value, is
set only during rsyslogd option processing */
David Lang
P.S. I think having the file named pmlastmsg but the intermal module
name being lastline is confusing, they should be related.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
