hi Jaren,
if you are using 'copy' instead of 'shift', I would modify the ruleset
a bit, and use additional context DOMAIN_$1 for storing the domain. In
the previous example, SENDER_$1 was always holding two lines, and
'shift' actions were required for extracting these lines into two
different variables. Since 'copy' always assings *both* lines (entire
event store) to a variable, the lines need to be stored into two
distinct contexts. Here is the updated ruleset (I haven't had time to
test it, and hopefully all modifications are correct):
type=single
ptype=RegExp
pattern=Mail\s(\d+):from=(.+@(.+))
desc=store sender address
action=create SENDER_$1 5; create DOMAIN_$1 5; add SENDER_$1 $2; add
DOMAIN_$1 $3
type=single
ptype=RegExp
pattern=Mail\s(\d+):subject=(.*)
context=SENDER_$1
desc=store subject
action=create SUBJECT_$1 5; add SUBJECT_$1 $2
type=single
ptype=RegExp
pattern=Mail\s(\d+):status=(.+)
context=SENDER_$1
desc=generate synthetic event for mail message
action=copy SENDER_$1 %email; copy DOMAIN_$1 %domain; \
copy SUBJECT_$1 %subject; delete SENDER_$1; delete DOMAIN_$1;
delete SUBJECT_$1; \
event
MAIL_$1_STATUS_$2_EMAIL_%{email}_DOMAIN_%{domain}_SUBJECT_%{subject}
Hope this helps,
risto
2017-06-13 14:33 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
> Hi,
>
> I´ve tested the rule and the problem that i see, it´s that "shift" command
> is not included in Sec 2.6. 2. I have changed to "copy" and it is working
> properly, i didnt see any incovenience.
>
> Thank you so much Risto!.Regards!.
>
> 2017-06-08 17:18 GMT+02:00 Jaren Peich <burkol...@gmail.com>:
>>
>> Hi,
>>
>> Thank you Risto! I was still blocked. I test it tomorrow and i´ll tell.
>>
>> Regards.
>>
>> 2017-06-07 13:26 GMT+02:00 Risto Vaarandi <risto.vaara...@gmail.com>:
>>>
>>> hi Jaren,
>>>
>>> I would recommend to divide the task into two parts:
>>> 1) normalization of log messages and the creation of one synthetic
>>> event from three raw log events
>>> 2) writing a thresholding rule for synthetic events generated during step
>>> 1
>>>
>>> As I understand from examples, each incoming e-mail generates three
>>> messages with the same numerical message ID. This message ID can be
>>> utilized during normalization, and here is a simple example that
>>> involves three Single rules:
>>>
>>> type=single
>>> ptype=RegExp
>>> pattern=Mail\s(\d+):from=(.+@(.+))
>>> desc=store sender address
>>> action=create SENDER_$1 5; add SENDER_$1 $2; add SENDER_$1 $3
>>>
>>> type=single
>>> ptype=RegExp
>>> pattern=Mail\s(\d+):subject=(.*)
>>> context=SENDER_$1
>>> desc=store subject
>>> action=create SUBJECT_$1 5; add SUBJECT_$1 $2
>>>
>>> type=single
>>> ptype=RegExp
>>> pattern=Mail\s(\d+):status=(.+)
>>> context=SENDER_$1
>>> desc=generate synthetic event for mail message
>>> action=shift SENDER_$1 %email; shift SENDER_$1 %domain; \
>>> shift SUBJECT_$1 %subject; delete SENDER_$1; delete SUBJECT_$1; \
>>> event
>>> MAIL_$1_STATUS_$2_EMAIL_%{email}_DOMAIN_%{domain}_SUBJECT_%{subject}
>>>
>>> The first and second rule match the e-mail and subject messages,
>>> respectively, and use contexts with a short lifetime to store relevant
>>> information extracted from these events. It is assumed that the
>>> subject message always comes after the e-mail message (as you can see,
>>> the second rule checks if SENDER_$1 context exists that has been
>>> created by first rule). The third rule reacts to the status message
>>> and creates a synthetic event from information extracted from all
>>> three events for the given message ID. For example, for the example
>>> events from your previous post, the following synthetic events are
>>> created:
>>>
>>> MAIL_1_STATUS_delivered_EMAIL_sec@sec.com_DOMAIN_sec.com_SUBJECT_Sec
>>> Configuration Rule
>>> MAIL_2_STATUS_delivered_EMAIL_pinker@sec.com_DOMAIN_sec.com_SUBJECT_Sec
>>> Configuration Rule
>>> MAIL_3_STATUS_delivered_EMAIL_plaster@sec.com_DOMAIN_sec.com_SUBJECT_Sec
>>> Configuration Rule
>>>
>>> MAIL_4_STATUS_delivered_EMAIL_plaster@panik.com_DOMAIN_panik.com_SUBJECT_Sec
>>> Configuration Rule
>>>
>>> MAIL_4_STATUS_Notdelivered_EMAIL_plaster@paniki.com_DOMAIN_paniki.com_SUBJECT_Sec
>>> Configuration Rule
>>>
>>> Once the normalization is done, you can use simple SingleWithThreshold
>>> rules for thresholding, without having a need to consider the
>>> "multiline event issue" each time you have to write a rule. For
>>> example, the following rule fires if 3 e-mails from the same domain
>>> with the same subject line have been delivered within 60 seconds:
>>>
>>> type=SingleWithThreshold
>>> ptype=RegExp
>>>
>>> pattern=MAIL_(\d+)_STATUS_delivered_EMAIL_(\S+?@\S+?)_DOMAIN_(\S+?)_SUBJECT_(.*)
>>> desc=3 e-mails from domain $3 with the same subject $4 have been
>>> delivered within 1 minute
>>> action=write - %s
>>> thresh=3
>>> window=60
>>>
>>>
>>> Hopefully these examples are helpful,
>>> risto
>>>
>>>
>>>
>>> 2017-06-06 13:45 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
>>> > Hi,
>>> >
>>> > I want to get all email from the same domain with the same subjects and
>>> > check if they have been delivered(status=delivered) with a thresh. The
>>> > problem is that the email log is split in different lines. The email
>>> > comes
>>> > with more lines but i´m only interested to get this(Log file lines).
>>> >
>>> > I have made 2 approches but i can´t validate domain variable in varmap
>>> > with
>>> > their aliaes to attach different context to a general context and
>>> > validate
>>> > all rule.
>>> > desc field must be "subject" variable.
>>> >
>>> > I´m using SEC 2.6.2 with strawberry perl.
>>> >
>>> > Log file(Sometimes could come disordered):
>>> >
>>> > Mail 1:from=s...@sec.com
>>> > Mail 1:subject=Sec Configuration Rule
>>> > Mail 1:status=delivered
>>> > Mail 2:from=pin...@sec.com
>>> > Mail 2:subject=Sec Configuration Rule
>>> > Mail 2:status=delivered
>>> > Mail 3:from=plas...@sec.com
>>> > Mail 3:subject=Sec Configuration Rule
>>> > Mail 3:status=delivered
>>> > Mail 4:from=plas...@panik.com
>>> > Mail 4:subject=Sec Configuration Rule
>>> > Mail 4:status=delivered
>>> > Mail 4:from=plas...@paniki.com
>>> > Mail 4:subject=Sec Configuration Rule
>>> > Mail 4:status=Notdelivered
>>> >
>>> >
>>> > ________________________________________________________________________________
>>> >
>>> >
>>> > First idea:
>>> >
>>> > type = Single
>>> > ptype = RegExp
>>> > continue = Takenext
>>> > context= Domain_$+{domain}
>>> > desc = $0
>>> > pattern = Mail\s(\d+)\:from\=(.*@(.*))
>>> > varmap= mid=1; email=2;domain=3;
>>> > action = alias Domain_$+{domain} HIT_Domain_$+{mid};fill
>>> > Domain_$+{domain}
>>> > $+{domain}
>>> >
>>> >
>>> > type = Single
>>> > ptype = RegExp
>>> > continue = Takenext
>>> > context= !Domain_$+{domain}
>>> > desc = $0
>>> > pattern = Mail\s(\d+)\:from\=(.*@(.*))
>>> > varmap= mid=1; email=2;domain=3;
>>> > action = create Domain_$+{domain} 86400; alias Domain_$+{domain}
>>> > HIT_Domain_$+{mid};
>>> >
>>> >
>>> >
>>> >
>>> > type=EventGroup2
>>> > ptype=regexp
>>> > pattern=Mail\s(\d+)\:subject\=(.*)
>>> > varmap= mid=1; subject=2
>>> > context = HIT_Domain_$+{mid}
>>> > thresh=2
>>> > count=alias Domain_$+{domain} Domain_Subject_$+{mid};
>>> > ptype2=regexp
>>> > pattern2=Mail\s(\d+)\:status\=delivered
>>> > varmap2= mid=1;
>>> > context2 = Domain_Subject_$+{mid} && HIT_Domain_$+{mid}
>>> > thresh=2
>>> > desc=Domain_$+{subject}
>>> > action= write - "Test String"
>>> > window=86400
>>> >
>>> >
>>> > _______________________________________________________________________________
>>> >
>>> > Second idea:
>>> >
>>> > type=EventGroup4
>>> > ptype = RegExp
>>> > continue = Takenext
>>> > context= Domain_$+{domain}
>>> > desc = $0
>>> > pattern = Mail\s(\d+)\:from\=(.*@(.*))
>>> > varmap= mid=1; email=2;domain=3;
>>> > count = alias Domain_$+{domain} HIT_Domain_$+{mid};
>>> > ptype2 = RegExp
>>> > continue2 = Takenext
>>> > context2= !Domain_$+{domain}
>>> > pattern2 = Mail\s(\d+)\:from\=(.*@(.*))
>>> > varmap2= mid=1; email=2;domain=3;
>>> > count2 = create Domain_$+{domain} 86400; alias Domain_$+{domain}
>>> > HIT_Domain_$+{mid};
>>> > ptype3=regexp
>>> > pattern3=Mail\s(\d+)\:subject\=(.*)
>>> > varmap3= mid=1; subject=2
>>> > context3 = HIT_Domain_$+{mid}
>>> > thresh3=2
>>> > count3=alias DOMAIN_$+{domain} Domain_Subject_$+{mid};
>>> > ptype4=regexp
>>> > pattern4=Mail\s(\d+)\:status\=delivered
>>> > varmap4= mid=1;
>>> > context4 = Domain_Subject_$+{mid} && HIT_Domain_$+{mid}
>>> > thresh4=2
>>> > desc=Domain_$+{subject}
>>> > action= write - "Test String"
>>> > window=86400
>>> >
>>> >
>>> > Any ideas?
>>> >
>>> > Thank you. Regards.
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
