In message <[email protected]>,
Tim Peiffer writes:
>I am interested in following Radius Detail records as a log input
>source. They are vertically aligned with attributes as 'tag = value'
>pairs. Are there any best practices for parsing such types of input
>with SEC? I tried using RegexpN, but found a number of the records were
>variable - some had other attributes that were being tracked, causing
>the number of lines to be somewhat inconsistent. In short, the parsing
>looked pretty gross. What words of wisdom can anyone impart for this
>kind of 'log' data?
>
>Mon Jan 24 23:33:38 2011
> User-Name = "JoeUser"
> NAS-Port = 70496256
> Service-Type = Framed-User
> Framed-Protocol = PPP
> Framed-IP-Address = 10.10.3.125
> Called-Station-Id = "10.21.217.82"
> Calling-Station-Id = "172.16.29.46"
> Acct-Status-Type = Start
> Acct-Delay-Time = 2
> Acct-Session-Id = "B1A039A3"
> Acct-Authentic = RADIUS
> NAS-Port-Type = Virtual
> Tunnel-Client-Endpoint = 172.16.29.46
> NAS-IP-Address = 192.168.249.28
> Timestamp = 1295933616
>
>Tue Jan 25 01:15:57 2011
> User-Name = "JoeUser"
> NAS-Port = 70496256
> Service-Type = Framed-User
> Framed-Protocol = PPP
> Framed-IP-Address = 10.10.3.125
> Called-Station-Id = "10.21.217.82"
> Calling-Station-Id = "172.16.29.46"
> Acct-Status-Type = Stop
> Acct-Delay-Time = 2
> Acct-Input-Octets = 13855207
> Acct-Output-Octets = 56621822
> Acct-Session-Id = "B1A039A3"
> Acct-Authentic = RADIUS
> Acct-Session-Time = 6139
> Acct-Input-Packets = 66890
> Acct-Output-Packets = 84696
> Acct-Terminate-Cause = User-Request
> NAS-Port-Type = Virtual
> Tunnel-Client-Endpoint = 172.16.29.46
> NAS-IP-Address = 192.168.249.28
> Timestamp = 1295939755
I would treat this using a classic line at a time parser. I answered a
similar problem for gathering java stack traces a few years back but
sadly I can't find the message in any of the online archives anymore.
It looks like every one of your events is started with a non space
character in column 1 and ends with a timestamp line.
I wrote this recipe for recognizing a multiline pattern when the
number of lines is unknown for a class I taught a couple of years ago.
It may give you an idea. Sadly the message referenced in this recipe
no longer exists.
=====
SEC supports the use of a counter on the end of the a number of the
pattern types such as SubStr3 or RegExp10. The number N makes the
pattern match against the last N lines in SEC's input buffer every
time a new line is read.
But you can have error messages in a multiline log message where the
number of lines changes depending on how the error was triggered.
For example you may want to find and report all
java.lang.OutOfMemoryError messages in a java stack dump and record
all the lines that led up to that. The problem is that the call stack
is dynamic and you may not know the number of elements in the stack
that is needed to set the N.
For example:
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at
org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1847)
at
org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:873)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1326)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1205)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
or
Exception in thread "main" java.lang.OutOfMemoryError
at java.lang.Thread.start(Native Method)
at com.ibm.ctg.server.ThreadManager.createObject(ThreadManager.java:345)
at com.ibm.ctg.server.ThreadManager.<init>(ThreadManager.java:131)
at com.ibm.ctg.server.ManagedResources.<init>(ManagedResources.java:106)
at com.ibm.ctg.server.JGate.main(JGate.java:895)
or
2008-04-23 06:18:50,418 [ERROR]
axis:org.apache.axis2.transport.http.AxisServlet unable to create new native
thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at
org.apache.commons.net.telnet.TelnetInputStream._start(TelnetInputStream.java:97)
at
org.apache.commons.net.telnet.TelnetClient._connectAction_(TelnetClient.java:104)
at org.apache.commons.net.SocketClient.connect(SocketClient.java:163)
at edu.internet2.hopi.dragon.DragonCSA.login(DragonCSA.java:60)
at net.es.oscars.pss.dragon.VlsrPSS.createPath(VlsrPSS.java:244)
at net.es.oscars.pss.PathSetupManager.create(PathSetupManager.java:91)
at
net.es.oscars.oscars.PathSetupAdapter.create(PathSetupAdapter.java:84)
at
net.es.oscars.oscars.OSCARSSkeleton.createPath(OSCARSSkeleton.java:438)
at
net.es.oscars.oscars.OSCARSMessageReceiverInOut.invokeBusinessLogic(OSCARSMessageReceiverInOut.java:174)
at
org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
To parse these out, use an idea discussed in:
http://sourceforge.net/mailarchive/message.php?msg_id=200510101206.j9AC6Srn029641%40mx1.cs.umb.edu.
Basically
you create a line at a time parser that recognizes the first line of a
dump and also recognized following lines. Note that the first line of
java stack dumps are not indented while the continuation lines are
indented.
The following rules collect all the java error messages that occur
within 10 seconds after an OutOfMemory error. First identify the
starter line by looking for a line that does not start with a space
when we are not already collecting a java stackdump using this rule:
type=single
rem = rule 1
desc=Identify first line of stack dump
ptype = regexp
pattern = ^[^\s].*java\.lang\.OutOfMemoryError
context = !collecting_java_stackdump
action = create collecting_java_stackdump 10 \
report collecting_java_stackdump \
/bin/mailx -s "java OOM error" adminuser; \
add $0 collecting_java_stackdump
This rule identifies the first line of a stack dump as a line that
does not have a space in the first column and contains the
OutOfMemoryError identifier. When we trigger this rule, it creates the
collecting_java_stackdump context which exists for 10 seconds. The
collecting_java_stackdump context has a lifetime of 10 seconds and
reports the contents of the context when it expires. As the last step,
the starting line is added to the collecting_java_stackdump context.
Next we collect all the additional lines in the log file using:
type=single
desc = accumulate continuation lines
rem = rule 2
ptype = regexp
pattern = .?
context = collecting_java_stackdump
action = add $0 collecting_java_stackdump
Note that this rule fires anytime the collecting_java_stackdump
context is valid, up to 10 seconds after the starter line is
recognized.
Now suppose we want to collect only the one error message and not all
the errors for a 10 second period. One way of doing this is to add
another rule before the first rule that clears (and reports) the
collecting_java_stackdump context. Given the format of the java error
(starter line has a non-whitespace character in column 1 and
continuation lines start with whitespace) you can add the rule:
type=single
desc = close current stackdump, start new one
rem = rule 0
continue=takenext
ptype = regexp
pattern = ^[^\s]
context = collecting_java_stackdump
action = obsolete collecting_java_stackdump
before rule 1. Rule 0 matches the start of a new stackdump only if we
are already collecting a stackdump. If we aren't matching a stackdump,
this rule is ignored. When this rule fires, it obsoletes (and thus
removes the collecting_java_stackdump context) but obsolete causes the
context expiration action to trigger. The expiration action reports
the collected stackdump. This rule has the continue keyword set to
takenext so that the event that triggered this rule is passed to the
following rule(s) to determine if the new stackdump should be
collected and reported.
===========
This doesn't discuss how to match a pattern where the interesting bit
is in the middle of the multiline sequence.
For example, suppose I only wanted to report an out of memory error
multi-line event that contains:
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1326
I create a single rule that matches this line if we are in the middle
of a stackdump, and this rule indicates that the line of interest is
present by creating a new context using the alias command. E.G.
type=single
desc = mark this as an interesting stackdump
rem = put this before rule 2
ptype = regexp
pattern = WebappClassLoader.loadClass.WebappClassLoader.java:1326
context = collecting_java_stackdump
action = alias collecting_java_stackdump it_is_interesting
and change rule 0 to be two rules, one that reports and one that
doesn't:
type=single
desc = close current stackdump, report it and start new one
rem = rule 0a
continue=takenext
ptype = regexp
pattern = ^[^\s]
context = collecting_java_stackdump && it_is_interesting
action = obsolete collecting_java_stackdump
and follow this with rule 0b
type=single
desc = only if rule 0a did not fire does collecting_java_stackdump exist
rem = so delete the context it since it's not interesting and start a new one
rem = rule 0b
continue=takenext
ptype = regexp
pattern = ^[^\s]
context = collecting_java_stackdump
action = delete collecting_java_stackdump
You can add more than one copy of the "mark this as an interesting
stackdump" single rule. Have each one create a unique context. Then
modify rule 0a so that it requires all the contexts to exist before it
triggers the report.
type=single
desc = close current stackdump, report it and start new one
rem = rule 0a
continue=takenext
ptype = regexp
pattern = ^[^\s]
context = collecting_java_stackdump && it_is_interesting && \
it_has_a_second_interesting_line && it_has_a_third_line_too
action = obsolete collecting_java_stackdump
So does this give you any ideas?
--
-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.
------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
