hi Przemysław,
I have provided my answers inline:
Then i tried this to incorporate it into Your proposal as a separate rule
> (inserted it right after the first rule).
> # create context for changed messadeid preserving login
> type=single
> ptype=regexp
> pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*relay=127.0.0.1.* queued as
> (?<MSGID2>\S+)\)
> context=MSGID_$+{MSGID}
> desc=set up a context for message ID $+{MSGID2}
> action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
> MSGID_$+{MSGID2} %{user}
>
> This seems to copy user data from old msgid to new msgid and make even
> multiple filter hops work ok.
>
> My question are:
> Is this optimal performance wise?
>
yes, it is probably the best way for addressing the problem. Contexts are
internally implemented as Perl hashes, and above three actions (copy,
create and fill) simply create a new hash entry and set it to a value from
an already existing entry. Since operations with hashes (such as looking up
an entry) work extremely fast in Perl, the actions don't have any
performance issues.
> Should i modify the action line to be like the one bellow, to get rid of
> now obsolote oryginal context, or will this create a race condition?
>
> action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
> MSGID_$+{MSGID2} %{user};delete MSGID_$+{MSGID}
>
If the old message ID is unnecessary and is no longer needed, I would
recommend to delete the context. Although individual context consumes very
little memory, contexts are checked periodically (by default each second)
by SEC for lifetime expiration, and inspecting thousands of unnecessary
contexts will involve extra cpu time. Therefore, deleting contexts that are
no longer needed helps to keep things optimal, even if you wouldn't notice
any performance difference :)
Also, there are no race conditions involved in the action list you have
defined. All actions in the list (copy, create, fill and delete) are
independent from each other, and the contexts they manipulate don't need to
exist throughout the execution of the entire action list. As for 'delete'
action, the context that is deleted does not necessarily need to exist, and
if it is not there, 'delete' is no-op (you would see a debug-level message
in the log about the missing context, though).
kind regards
risto
>
> Regards
> Przemyslaw Orzechowski
>
> W dniu 12.06.2018 o 21:12, Risto Vaarandi pisze:
>
> hi Przemysław,
>
> for addressing this problem, you could divide it into several independent
> subtasks:
> 1) memorize the mapping from msgid to username, so that each mapping would
> exist for a certain amount of time
> 2) when a bounce event appears in the log, generate a synthetic event for
> the bounce, so that msgid->username mapping is used for including the user
> name in the synthetic event
> 3) count and threshold synthetic events, and generate an alarm if more
> than 10 events have been seen in 1 hour for the given user name
>
> To illustrate how this could work, I've put together a small rulebase of
> three rules, with each rule corresponding to one of the subtasks above:
>
> # establish msgid->username mapping
>
> type=single
> ptype=regexp
> pattern=postfix.*\]: (?<MSGID>\S+): .* sasl_username=(?<LOGIN>\S+)
> desc=set up a context for message ID $+{MSGID}
> action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
>
> # generate a synthetic bounce event which contains the user name
> # (username is obtained through msgid->username mapping)
>
> type=single
> ptype=regexp
> pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*dsn=5.*, status=bounced
> context=MSGID_$+{MSGID}
> desc=create a synthetic event for a bounce event with a username
> action=copy MSGID_$+{MSGID} %user; event BOUNCE_FOR_USER_%{user}_MSGID_
> $+{MSGID}
>
> # count and threshold synthetic events generated above
>
> type=singlewiththreshold
> ptype=regexp
> pattern=BOUNCE_FOR_USER_(?<LOGIN>\S+)_MSGID_(\S+)
> desc=Too many bounces for user $+{LOGIN} during 1 hour
> action=write - %s
> thresh=10
> window=3600
>
> For recording the mappings, the first rule simply uses a context for each
> msgid, where the message ID is used as a context name, and user name is
> written into context's event store with the 'fill' action. Each context
> exists for 1 hour, but you can increase the mapping lifetime by using a
> larger value instead of 3600 seconds for the 'create' action.
>
> The second rule converts the original bounce event into a synthetic event
> which contains the user name. Note that the user name is taken from the
> context for the given msgid with the 'copy' action (if the context does not
> exist, the rule would not match the original bounce event due to 'context'
> field, and the synthetic event is not created).
>
> Finally, the third rule implements event counting which is done on user
> name basis. If you would like to do it per each unique (username,msgid)
> tuple, you can include the msgid into the 'desc' field (the regular
> expression would also need to be modified for setting the $+{MSGID}
> variable).
>
> Hope this helps,
> risto
>
> 2018-06-12 17:55 GMT+03:00 Przemysław Orzechowski <
> przemek.orzechow...@makolab.com>:
>
>> Hi
>>
>>
>> I have successfully created 2 rules that mach lines i want to act on but
>> I'm struggling in finding a way to correlate them.
>>
>> First description what i want to get
>>
>> Generate an event for a threshold ie 10 events in 1 hour the problem is i
>> don't quite get how to count them correctly
>>
>> First is a line that has username and mailid for example:
>>
>> Jun 3 06:46:13 server1 postfix/smtpd[3268]: 626C5802295:
>> client=unknown[8.8.8.8], sasl_method=LOGIN, sasl_username=
>> lo...@some.domain.com
>>
>> Im maching it with pattern
>>
>> pattern=postfix.*\]: (?<MSGID>\S+): .* sasl_username=(?<LOGIN>\S+)
>>
>> the threshold should be for unique LOGINS and should count events mached
>> by the second rule ie lines like these:
>>
>>
>> Jun 3 06:46:14 server1 postfix/smtp[23808]: 626C5802295: to=<
>> somb...@somewhere.com>, relay=somewhere.com[9.9.9.9]:25, delay=1,
>> delays=1/0.03/21/20, dsn=5.0.0, status=bounced (host somewhere.com[9.9.9.9]
>> said: 550-The mail server could not deliver mail to somb...@somewhere.com.
>> 550-The account or domain may not exist, they may be blacklisted, or
>> missing 550 the proper dns entries. (in reply to RCPT TO command))
>>
>> Im maching this line with pattern
>>
>> pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*dsn=5.*
>>
>> the MSGID is identical in both lines
>>
>> Im trying to count the bounces and execute an action if there are too
>> many of them for single sasl_username in a time period
>>
>> Btw MSGID can repeat themselvs in longer preiods of time but should be
>> unique within the treshold period.
>>
>> I would be very thankfull for any help
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
