2015-07-02 21:44 GMT+03:00 Rajesh M <[email protected]>:
> Hi Risto,
>
>
> Thank You once again for your kind comment.
>
>
> Actually, my script my.pl is used for taking out the syslog events from X
> timestamp [intial time] to Y timestamp [New time after adding some
> seconds]. This intial timestamp is determined by searching for particular
> pattern [ALARM RAISE] of event in syslog.Finally printing all the syslog
> events from X to Y timestamps.
>
>
> I wanted to pipe this script output to SEC again, I have a second rule in
> my .conf file which can match event in this script output. I'm implementing
> all beacause my inputs are non realtime events.
>
>
>
> Please try to make some time avaliable for yourself to check the below
> details of my project.
>
>
> I am working on a project for log correlation engine where we want to have
> tool which can check if the issue reported is already known or not and
> hence I thought of using SEC for this. The below is the abstract info about
> the project and my requirement.
>
>
>
> *Scenario:*
>
> Basically our idea is to have a database of signatures ( conf files etc )
> which consists of issues found in our product ( means set of events in
> various log files which point to particular issue ) and we just want to run
> this tool on new issue/error reported on our product and see if this issue
> was already existing.
>
>
>
> *Example: *
>
>
>
> We have two logfiles, logfile1 and logfile2 and we want to know is a
> pattern is seen in these logs which was reported in earlier known cases
>
>
>
> *Logfile1 snippet:*
>
> Aug 21 21:16:00.168055 info CFPU-1 rsyslogd: file /srv/Log/log/syslog was
> successfully rotated
>
> Aug 21 21:16:00.376903 info CFPU-0 ResourceMeas: COMMUTIL INFO [Ag]
> Measurement job created, data for all objects is reported by default
>
> Aug 21 21:16:01.319159 info CFPU-1 crond[30812]: (root) CMD (nice
> /opt/nokiasiemens/bin/timestamp.sh)
>
>
>
> *Logfile2 snippet:*
>
> Aug 21 21:16:14.35245 info tid=0x93801200 <== set_state_and_signal -
> tx_state TS_ACTIVE -> TS_PROCESSING
>
> Aug 21 21:17:14.35540 info tid=0x77001200 <== set_state_and_signal -
> tx_state TS_PROCESSING -> TS_SEARCHING_IN_PROCESSING
>
>
>
>
>
> *Now my project is to check the Logfile1 for following event*:
>
> rsyslogd: file /srv/Log/log/syslog was successfully rotated
>
>
>
> *And this should be followed by another in logfile2:*
>
> 35245 info tid=0x93801200 <== set_state_and_signal - tx_state TS_ACTIVE ->
> TS_PROCESSING
>
>
>
> And both of these event should have occurred in *a span of 1 minute* for
> correlation to succeed.
>
>
>
> Limitation is that we will not be using these files at runtime means these
> files ar not having same time as of system ( basically timestamps in
> logfiles are not same as system time )
>
>
>
> *So kindly help me with below*:
>
> 1) 1) Is SEC the right tool for this usecase ?
>
It depends entirely on your requirements. Since sec is a real-time event
correlator, the only way to use its full functionality for past events is
to do a full replay which covers the time span of the entire log.
Therefore, if your log covers 24 hours (say, from Mar 13 00:00:00 to Mar 14
00:00:00), you would also need 24 hours for the replay. Obviously, this is
not convenient for longer event logs, and would work only for logs that
cover shorter time frames. Of course, one could implement various
workarounds to increase the processing speed (such as identifying a
relevant event and replaying only events which follow this event within 1-2
minutes), but such workarounds often limit sec to particular event
correlation ruleset only, and might not be applicable for other rulesets.
If you are only interested in finding pairs of events (i.e., you want to
verify if event A is followed by B within 60 seconds), it is probably much
simpler to write a 50-100 line Perl/Python script for that particular
problem. On one hand, that script would only work for detecting specific
event pairs, but on the other hand, you wouldn't need to replay events, but
can process logs that cover several hours at the full speed of your cpu and
disk.
In other words, it only makes sense to use sec if the replay of past events
is acceptable for you. If it is not feasible, a custom script would be a
much better solution, especially if you want to check just few event
correlation scenarios which are not very complex.
2) 2) Are there any examples to looks for these ?
>
I am not aware of any specific examples for doing full correlation of past
events. Also, these issues have only been briefly mentioned in the mailing
list before -- probably because sec is explicitly designed for correlating
real-time events.
> 3) 3) Do we have detailed knowledge base for all
> features/keyword/functionality of SEC with examples ?
>
If replay of events is acceptable for you, they are just like regular
real-time events for sec, and therefore everything in official
documentation and sec papers is fully applicable. For gentle introduction
to sec, I'd recommend to check the following papers by John Rouillard and
David Lang:
http://www.cs.umb.edu/~rouilj/sec/sec_paper.pdf
https://www.usenix.org/system/files/login/articles/09_lang-online.pdf
You can find links to these papers in the sec home page in the
"Documentation" section. This section also contains pointers to some other
sec papers.
kind regards,
risto
>
>
> Thanks in advance and look forward for your advice.
>
>
> Thanks & Regards,
>
> Karthik
>
> On Thu, Jul 2, 2015 at 4:07 PM, Risto Vaarandi <[email protected]>
> wrote:
>
>> ...to complement my previous mail, I think it is simpler to set up
>> dedicated perl scripts for preparing input for sec, rather than trying to
>> incorporate data preprocessing into sec rules.
>>
>> Here are two perl scripts which follow the example Mark has already
>> provided. Firstly, generate_timestamp.pl looks like follows (note that
>> it currently works for timestamps like "2015 Jun 20 12:00:00" and "Jun 21
>> 12:13:41", and for other formats you need to adjust the timestamp parsing
>> regular expression):
>>
>> #!/usr/bin/perl -w
>> #
>> # generate_timestamp.pl
>> # Prepend 'seconds since epoch' to each input line
>>
>> use Time::Local;
>>
>> %months = ( 'Jan' => 0, 'Feb' => 1, 'Mar' => 2, 'Apr' => 3,
>> 'May' => 4, 'Jun' => 5, 'Jul' => 6, 'Aug' => 7,
>> 'Sep' => 8, 'Oct' => 9, 'Nov' => 10, 'Dec' => 11 );
>>
>> @time = localtime(time());
>>
>> $year = $time[5] + 1900;
>> $month = $time[4];
>>
>> while (<STDIN>) {
>> if
>> (/^(?:(\d{4})\s+)?([A-Z][a-z]{2})\s+(\d{1,2})\s+(\d{2}):(\d{2}):(\d{2})/) {
>> $y = defined($1)?$1:$year;
>> $m = exists($months{$2})?$months{$2}:$month;
>> $time = timelocal($6, $5, $4, $3, $m, $y);
>> print "$time ", $_;
>> }
>> }
>>
>>
>> And then the replay_events.pl script:
>>
>> #!/usr/bin/perl -w
>> #
>> # replay_events.pl
>> # Replay sorted events generated by generate_timestamp.pl
>>
>> select STDOUT;
>> $| = 1;
>>
>> while (<STDIN>) {
>>
>> if ($_ !~ /^(\d+) (.*)/) { next; }
>>
>> if (!defined($previous)) {
>> print $2, "\n";
>> $previous = $1;
>> next;
>> }
>>
>> $d = $1 - $previous;
>> if ($d < 0) { next; }
>> sleep($d);
>> print $2, "\n";
>> $previous = $1;
>> }
>>
>>
>> After having those scripts implemented, replaying past events becomes a
>> matter of simple UNIX pipeline. For example, the following pipeline joins
>> /var/log/messages and /var/log/secure into a single event stream and
>> replays this to sec:
>>
>> cat /var/log/messages /var/log/secure | ./generate_timestamp.pl | sort |
>> ./replay_events.pl | /usr/bin/sec --conf=test.conf --input=- --notail
>>
>> Also, I would suggest to include any database queries into a dedicated
>> preprocessing script which precedes sec in the UNIX pipeline.
>>
>> To summarize, I strongly believe that the clear separation of data
>> preprocessing from sec-based event correlation is the best solution for
>> you, in order to keep your configuration manageable and efficient.
>>
>> hope this helps,
>> risto
>>
>> 2015-07-01 13:15 GMT+03:00 Rajesh M <[email protected]>:
>>
>>> Hi Risto,
>>>
>>> I am implementing a perl script which basically accepts the input file
>>> and time in sec, after search for particular pattern and take out the time
>>> stamp and adds input time to that, generates "New Time stamp". Later the
>>> output is syslog events from input file between Original TS and New TS.
>>>
>>> my .conf file:
>>> ------------------
>>> type=Single
>>> ptype=RegExp
>>> pattern="ALARM RAISE 70307" [Very first rasie event in
>>> alarm.log]
>>> desc=$0
>>> action=spawn /var/test/my.pl
>>>
>>> type=Single
>>> ptype=RegExp
>>> pattern=.*logf started
>>> desc=Matched event
>>> action=write /home/test.out
>>>
>>>
>>> Basically I am trying to trigger the scrpit based on my 1st rule match
>>> and wants to fed the script output to SEC for 2nd rule match.
>>>
>>> Whenever I was executing this .conf, the script was calling and it
>>> wouldn't goto 2nd rule and also the script was exectuing how many number of
>>> times the 1st rule matches.Please suggest what is the correct way of doing
>>> this operation in SEC.
>>>
>>> Thanks & Regards,
>>> Karthik
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Don't Limit Your Business. Reach for the Cloud.
>>> GigeNET's Cloud Solutions provide you with the tools and support that
>>> you need to offload your IT needs and focus on growing your business.
>>> Configured For All Businesses. Start Your Cloud Today.
>>> https://www.gigenetcloud.com/
>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>>>
>>
>
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
