Hi,
Many Thanks for your help. i will check the code in those days and ill tell
you.
Regards. :)
2015-10-07 12:44 GMT+02:00 Risto Vaarandi <risto.vaara...@gmail.com>:
> hi Jaren,
> note that the 'add' function does not numerically add a value to the
> context event store, but it rather appends textual line(s) to context event
> store. Since the context event store holds textual data only, you can't use
> it in the way you would use the stack in Forth language.
> If you wish to implement byte summing, it is sufficient to implement
> couple of perl code snippets into your ruleset. Here is one example which
> you might use for addressing the problem:
>
> type=single
> ptype=regexp
> pattern=user (?<user>[\w.-]+) bytes (?<bytes_uploaded>\d+)
> varmap=data_upload
> continue=takenext
> desc=parse data upload events
> action=none
>
> type=Single
> ptype=Cached
> pattern=data_upload
> continue=TakeNext
> context=!BYTES_$+{user}
> desc=create a context for new user $+{user}
> action=create BYTES_$+{user} 300; add BYTES_$+{user} 0
>
> type= Single
> ptype=Cached
> pattern=data_upload
> continue=TakeNext
> context=BYTES_$+{user}
> desc=add $+{bytes_uploaded} to the upload counter of user $+{user}
> action=pop BYTES_$+{user} %prev; \
> lcall %sum %prev $+{bytes_uploaded} -> ( sub { $_[0] + $_[1] } ); \
> add BYTES_$+{user} %sum;
>
> type=Single
> ptype=Cached
> pattern=data_upload
> desc=check if user $+{user} has uploaded more than 100000 bytes
> continue=TakeNext
> context=BYTES_$+{user}
> action=pop BYTES_$+{user} %sum; lcall %o %sum -> ( sub { $_[0] > 100000 }
> ); \
> if %o (write - User $+{user} has uploaded too many bytes (%sum)); \
> add BYTES_$+{user} %sum
>
>
> Hope this helps,
> risto
>
> 2015-10-07 12:27 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
>
>> Hi,
>>
>> Sorry, i couldn´t answer before but i thought something like that:
>>
>> Here i can detect and create a context for each user.
>>
>> type = Single
>>
>> ptype = Cached
>>
>> desc = $0
>>
>> continue = TakeNext
>>
>> context = !BYTES_$+{user}
>>
>> pattern = pattern
>>
>> action = create BYTES_$+{user} 300;
>>
>> We check that the context exists and we add bytes in each context:
>>
>>
>> type = Single
>>
>> ptype = Cached
>>
>> desc = $0
>>
>> continue = TakeNext
>>
>> context = BYTES_$+{user}
>>
>> pattern = pattern_que_necesites
>>
>> action = *add* BYTES_$+{user} $bytes_uploaded
>>
>>
>> Finally i pop the result:
>>
>>
>> type = Single
>>
>> ptype = Cached
>>
>> desc = $+{user}
>>
>> continue = TakeNext
>>
>> context = BYTES_$+{user}
>>
>> pattern = pattern
>>
>> action = *pop* BYTES_$+{user} %bytes_uploaded_$+{user} ; \
>>
>>
>> I don´t know how read the bytes added to the context. is it possible?
>>
>>
>> Regards.
>>
>>
>>
>> 2015-09-30 14:34 GMT+02:00 Risto Vaarandi <risto.vaara...@gmail.com>:
>>
>>> hi Jaren,
>>> for using a separate window for each individual user, you can't use
>>> SingleWithThreshold operations, since their internal counters reflect event
>>> count, and the occurrence of a matching events always adds 1 to the
>>> counter. Since you want to implement event counters where values greater
>>> than 1 can be added, the rules need some additional Perl code. For
>>> instance, the following ruleset addresses the counting scheme for users
>>> Korsakof, Bob and Alice, and triggers an alarm if one of those users
>>> uploads more than 100MB within 900 seconds (15 minutes):
>>>
>>> # Clean entries from the %bytes hash table which are older than 15
>>> minutes
>>>
>>> type=Calendar
>>> time=* * * * *
>>> desc=clean old entries from the bytes hash table
>>> action=lcall %o -> ( sub { my($user); my($time) = time(); \
>>> foreach $user (keys %bytes) { \
>>> if ($bytes{$user}->[-1]->[0] < $time - 900) \
>>> { delete $bytes{$user}; } } } )
>>>
>>> # Suppress the user if he/she is not Korsakof, Bob or Alice
>>>
>>> type=Suppress
>>> ptype=RegExp
>>> pattern=^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ (?!(?:Korsakof|Bob|Alice)
>>> )\S+ \d+
>>>
>>> # Use the %bytes hash table for memorizing each upload event for the
>>> user,
>>> # and record the time and bytecount for each upload. When the upload
>>> event
>>> # occurs, drop all previous upload events which are older than 15
>>> minutes,
>>> # and sum the bytecounts of remaining uploads. If the sum is greater than
>>> # 100MB, produce an alarm, and suppress further alarms for 2 hours.
>>>
>>> type=Single
>>> ptype=RegExp
>>> pattern=^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ ([\w.@-]+) (\d+)
>>> context=$1 $2 -> ( sub { my($time) = time(); my($sum) = 0; \
>>> push @{$bytes{$_[0]}}, [ $time, $_[1] ]; \
>>> while ($bytes{$_[0]}->[0]->[0] < $time - 900) \
>>> { shift @{$bytes{$_[0]}}; } \
>>> map { $sum += $_->[1] } @{$bytes{$_[0]}}; \
>>> return ($sum > 104857600); } ) \
>>> && !SUPPRESS_ALARMS_FOR_$1
>>> desc=User $1 has transmitted too many bytes
>>> action=write - %s; create SUPPRESS_ALARMS_FOR_$1 7200
>>>
>>>
>>> The most complex part of the above ruleset is the context expression of
>>> the last rule which employs the %bytes hash table for memorizing all
>>> uploads for the last 15 minutes. As in my previous ruleset example, the
>>> hash table keys are user names, but this time each user name points to a
>>> list of upload events. Each upload event is in turn memorized as a tuple
>>> (time_of_upload, bytecount). In other words, $bytes{"Bob"}->[0]->[0] refers
>>> to the time of the earliest upload event for user Bob, while
>>> $bytes{"Alice"}->[0]->[1] refers to the bytecount of the earliest upload
>>> event for user Alice. The Perl function in the context expression uses the
>>> time and bytecount fields for dropping events older than 15 minutes first,
>>> and then summing up the bytes of remaining events and comparing the sum to
>>> 104857600 (100MB). This ensures that you are always doing thresholding for
>>> the last 15 minutes.
>>>
>>> If you wish to filter relevant user names in a different way (e.g., you
>>> want to exclude few user names and apply thresholding for the rest), you
>>> need to modify the regular expression of the second rule accordingly.
>>>
>>> Hope this helps,
>>> risto
>>>
>>> 2015-09-30 14:20 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> Thanks for your help. I was thinking and i can not block user during 2H
>>>> without alerts or in any time.
>>>> I was thinking and how can i add a temporaly window?.Can i change the
>>>> alert to SingleWithThreshold and just add a window variable without
>>>> thresh?Could i limit the number of users?
>>>>
>>>> The idea its like that: User Jaren has uploaded 100MB in 15 min.
>>>>
>>>>
>>>>
>>>> Regards.
>>>>
>>>> 2015-09-30 13:19 GMT+02:00 Jaren Peich <burkol...@gmail.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks for your help. I was thinking and i can not block user during
>>>>> 2H without alerts or in any time.
>>>>> I was thinking and how can i add a temporaly window?.Can i change the
>>>>> alert to SingleWithThreshold and just add a window variable without
>>>>> thresh?Could i limit the number of users?
>>>>>
>>>>> The idea its like that: User Jaren has uploaded 100MB in 15 min.
>>>>>
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> 2015-09-29 16:19 GMT+02:00 Risto Vaarandi <risto.vaara...@gmail.com>:
>>>>>
>>>>>> hi Jaren,
>>>>>> you could try the following simple ruleset and see if this fits your
>>>>>> needs:
>>>>>>
>>>>>> type=Calendar
>>>>>> time=0 * * * *
>>>>>> desc=drop byte counters
>>>>>> action=lcall %o -> ( sub { %bytes = () } )
>>>>>>
>>>>>> type=Single
>>>>>> ptype=RegExp
>>>>>> pattern=^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ ([\w.@-]+) (\d+)
>>>>>> context=$1 $2 -> ( sub { ($bytes{$_[0]} += $_[1]) > 10000 } ) && \
>>>>>> !SUPPRESS_ALARMS_FOR_$1
>>>>>> desc=User $1 has transmitted too many bytes
>>>>>> action=write - %s; create SUPPRESS_ALARMS_FOR_$1 7200
>>>>>>
>>>>>> This ruleset employs the Perl hash table %bytes for keeping track of
>>>>>> user data transfers. The keys to %bytes hash table are user names, and
>>>>>> the
>>>>>> value which corresponds to each user name is the number of bytes
>>>>>> transmitted.
>>>>>>
>>>>>> Since you didn't mention in which particular time window you would
>>>>>> like to accomplish upload tracking, I have chosen a fixed window of 1
>>>>>> hour
>>>>>> which is applied to all users. The purpose of the first rule is to drop
>>>>>> byte counters for all users once every hour. If you wish to accomplish
>>>>>> the
>>>>>> monitoring in a larger window, it is easy to adjust the 'time' field of
>>>>>> the
>>>>>> Calendar rule accordingly (the field takes a value in crontab syntax).
>>>>>>
>>>>>> The second rule matches individual upload events with the regular
>>>>>> expression
>>>>>>
>>>>>> ^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ ([\w.@-]+) (\d+)
>>>>>>
>>>>>> which sets $1 to user name and $2 to transmitted bytes. I have
>>>>>> assumed that legal characters for user names are letters, digits,
>>>>>> underscores, dots, dashes and @-signs, and therefore I've used ([\w.@-]+)
>>>>>> for matching the username. If you want your username to be just any
>>>>>> sequence of non-whitespace characters, you can use (\S+) instead. In
>>>>>> order
>>>>>> to track the number of bytes user has uploaded, the rule uses the
>>>>>> following
>>>>>> context expression
>>>>>>
>>>>>> $1 $2 -> ( sub { ($bytes{$_[0]} += $_[1]) > 10000 } ) && \
>>>>>> !SUPPRESS_ALARMS_FOR_$1
>>>>>>
>>>>>> which consists of two operands joined by logical AND (&&).
>>>>>>
>>>>>> The first operand '$1 $2 -> ( sub { ($bytes{$_[0]} += $_[1]) > 10000
>>>>>> } )' is a call to a precompiled perl function which simply adds bytes for
>>>>>> the current upload to the total of the given user. Note that the
>>>>>> variables
>>>>>> $1 (user name) and $2 (bytes) serve as input parameters for the function.
>>>>>> Also, the function checks if the resulting user total exceeds the
>>>>>> threshold
>>>>>> (I have used 10000 bytes for threshold which probably needs replacement
>>>>>> by
>>>>>> a more reasonable value). The second operand of the context expression
>>>>>> !SUPPRESS_ALARMS_FOR_$1 is true if the context SUPPRESS_ALARMS_FOR_$1
>>>>>> does
>>>>>> not exist (which is the case if we haven't generated alarm for the given
>>>>>> user within the last 2 hours -- see the following explanation). If both
>>>>>> operands are true (in other words, the upload threshold is violated and
>>>>>> we
>>>>>> haven't produced an alert about the user in last 2h), the rule will
>>>>>> write a
>>>>>> warning string to standard output with the 'write' action. Also, after
>>>>>> generating the alarm, the rule creates the SUPPRESS_ALARMS_FOR_$1 context
>>>>>> with a lifetime of 7200 seconds. Since the context will stay around for 2
>>>>>> hours, it suppresses further alarms for the same user for this time
>>>>>> frame.
>>>>>>
>>>>>> Hope this helps,
>>>>>> risto
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-09-29 11:21 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Sorry for my English and thanks for the perl library. Thanks for
>>>>>>> accepting me in mailing list. :)
>>>>>>>
>>>>>>> I have to program a code with SEC to detect a quantity of bytes
>>>>>>> uploaded to the web from one user with a limit. I dont know how to sum
>>>>>>> the
>>>>>>> quantity of bytes that comes from each proxy log line till reach the
>>>>>>> limit
>>>>>>> and generate an alert.
>>>>>>>
>>>>>>> Log Line
>>>>>>>
>>>>>>> Time: Method "Url" User Bytes
>>>>>>> 29/09/2015 10:14:POST "www.google.com" Korsakof 1250
>>>>>>>
>>>>>>> Thank you. Regards.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Simple-evcorr-users mailing list
>>>>>>> Simple-evcorr-users@lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
------------------------------------------------------------------------------
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
