On 12/12/2011 04:01 PM, Alberto Cortón wrote: > Hi, > > I would like to know if any of you have used SEC for normalizing log data. My > first approach to this was to generate normalized events like this: > > action = event > 'TIME=$1:::CODE=$3:::SRC_IP=$4:::SRC_PORT=$5:::DST_IP=$6:::DST_PORT=$7:::IFACE=$8:::PROTOCOL=$9:::ACTION=$10' > > and use a single pattern for generating the alerts > > pattern = > TIME=(.+?):::CODE=(.+?):::SRC_IP=(.+?):::SRC_PORT=(.+?):::DST_IP=(.+?):::DST_PORT=(.+?):::IFACE=(.+?):::PROTOCOL=(.+?):::ACTION=(.+?) > > But this list could grow up to twenty or more parameters, so you would need a > VERY large pattern for capturing the normalized events. Moreover, due to not > all the devices log the same info (sometimes there is no interface > information, for example) I would need to add several empty fields on almost > any action. > > Any thoughts or suggestions? >
...to add another idea -- if you want to run a very fast normalization on logs with multi-line events, you could also take advantage of the LogPP (Log PreProcessor) utility at http://logpp.sourceforge.net. I wrote it some years ago for fast processing and flatfile -> syslog conversion. Although syslog-ng also allows for converting flatfile logs to syslog, they don't support couple of things that logpp can do for you -- multi-line to single-line conversion, and the ability to extract input file names from input events. hth, risto > Regards, > > > ------------------------------------------------------------------------------ > Learn Windows Azure Live! Tuesday, Dec 13, 2011 > Microsoft is holding a special Learn Windows Azure training event for > developers. It will provide a great way to learn Windows Azure and what it > provides. You can attend the event by watching it streamed LIVE online. > Learn more at http://p.sf.net/sfu/ms-windowsazure > _______________________________________________ > Simple-evcorr-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > ------------------------------------------------------------------------------ Systems Optimization Self Assessment Improve efficiency and utilization of IT resources. Drive out cost and improve service delivery. Take 5 minutes to use this Systems Optimization Self Assessment. http://www.accelacomm.com/jaw/sdnl/114/51450054/ _______________________________________________ Simple-evcorr-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
