Risto, That's important advice from a performance perspective, so thank you for that. I would have missed it and suffered as a result.
If we decide to run SEC like a daemon, as the data is generated (which we might do), then all that stale data would have definitely caused us problems (the callid DOES get re-used, as you surmised). I'm sure I can figure out how to put a handler in to drop everything relating to a call once it's done. I'll do some reading (I always learn new things as I re-read the docs, so that's time well spent). I like the option to use Perl to handle the counter; I'll see how the performance does and decide based on that. Thanks as always for the help. Jim -- Jim Van Meggelen ClearlyCore Inc. +1-416-639-6001 (DID) +1-877-253-2716 (Canada) +1-866-644-7729 (USA) +1-416-425-6111 x6001 jim.vanmegge...@clearlycore.com [ http://www.clearlycore.com/ | http://www.clearlycore.com ] Asterisk: The Definitive Guide FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] > From: "Risto Vaarandi" <risto.vaara...@gmail.com> > To: "Jim Van Meggelen" <jim.vanmegge...@clearlycore.com> > Cc: "simple-evcorr-users" <simple-evcorr-users@lists.sourceforge.net> > Sent: Sunday, 24 September, 2023 04:35:50 > Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context > hi Jim, > many thanks for the clarification! I have a better understanding now about the > nature of the problem. > Because you are doing offline processing of an already existing log file, > removing old contexts or old hash table keys (recommendation from the end of > my > post) might not be so important, provided that call IDs are not reused for > different calls in the processed log file, and the number of call IDs is not > very large. That would keep the solution a bit more simple. > kind regards, > risto > Kontakt Jim Van Meggelen (< [ mailto:jim.vanmegge...@clearlycore.com | > jim.vanmegge...@clearlycore.com ] >) kirjutas kuupäeval L, 23. september 2023 > kell 22:11: >> Risto, >> I cannot change the timestamps, as they are coded in the files I'm working >> with >> (old data). Going forward, I also cannot guarantee that I will have control >> over the source data, and it is likely that it will have a similar >> granularity. >> Finally, there's no need for that level of granularity downstream; it is only >> necessary that two events with the same timestamp are sequenced so that it is >> clear which happened first. >> It does seem a bit kludgy, but if I try to reduce the kludge in my SEC file, >> I >> am almost certain to have to introduce kludge somewhere else, which is worse >> (this way I only have SEC to maintain, rather than additional technology). >> This way, I can feed relevant log files to SEC from any source, and trust >> that >> I'll be able to handle them, so long as they follow standard Asterisk log >> format. >> Thanks again for your generous advice. >> Jim >> -- >> Jim Van Meggelen >> ClearlyCore Inc. >> +1-416-639-6001 (DID) >> +1-877-253-2716 (Canada) >> +1-866-644-7729 (USA) >> +1-416-425-6111 x6001 >> [ mailto:jim.vanmegge...@clearlycore.com | jim.vanmegge...@clearlycore.com ] >> [ http://www.clearlycore.com/ | http://www.clearlycore.com ] >> Asterisk: The Definitive Guide >> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: >> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | >> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] >>> From: "Risto Vaarandi" < [ mailto:risto.vaara...@gmail.com | >>> risto.vaara...@gmail.com ] > >>> To: "Jim Van Meggelen" < [ mailto:jim.vanmegge...@clearlycore.com | >>> jim.vanmegge...@clearlycore.com ] > >>> Cc: "simple-evcorr-users" < [ >>> mailto:simple-evcorr-users@lists.sourceforge.net | >>> simple-evcorr-users@lists.sourceforge.net ] > >>> Sent: Saturday, 23 September, 2023 03:51:02 >>> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context >>> hi Jim, >>> the solutions from my previous post were provided without knowing the root >>> cause >>> of the issue. If the main reason for the problem is the nature of the >>> timestamps in the logs (that is, they are provided with an accuracy of a >>> second), I would recommend using a different logging scheme with >>> high-resolution timestamps. For example, as David has mentioned in his post, >>> using syslog servers with RFC5424-based syslog protocol ( [ >>> https://datatracker.ietf.org/doc/html/rfc5424 | >>> https://datatracker.ietf.org/doc/html/rfc5424 ] ) provides such timestamps >>> out-of-the-box. I am not sure whether you can change the logging scheme for >>> these particular application messages, but if you can, these timestamps are >>> highly useful for many purposes like the investigation of past system faults >>> and security incidents. >>> kind regards, >>> risto >>>> hi Jim, >>>> let me provide some suggestions how to accomplish this task. First, you >>>> could >>>> utilize the context based approach that you have described in your post. In >>>> that case, the numeral that has been retrieved from the context with the >>>> 'pop' >>>> action needs to be incremented, and you can do it with the 'lcall' action >>>> that >>>> invokes the relevant Perl code. Here is one example (I have added several >>>> of >>>> extra actions into the rule for initializing the context): >>>> type=single >>>> ptype=regexp >>>> pattern=call_id (\S+) >>>> desc=call id $1 >>>> action=exists %e call_uniqueid_seq_$1; \ >>>> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \ >>>> pop call_uniqueid_seq_$1 %sequence_num; \ >>>> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \ >>>> add call_uniqueid_seq_$1 %sequence_num >>>> The first two actions check if the context call_uniqueid_seq_* for the >>>> given >>>> call exists, and if not, 0 is written into that context to initialize it: >>>> exists %e call_uniqueid_seq_$1; \ >>>> if %e ( none ) else ( add call_uniqueid_seq_$1 0); >>>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are >>>> identical to the actions from your email. >>>> The 'lcall' action (4th action in the above rule) increments the counter >>>> value >>>> retrieved from the context by executing the following Perl one-line >>>> function: >>>> sub { $_[0] + 1 } >>>> That function takes the value of its parameter (that is, %sequence_num) and >>>> returns an incremented value which is stored back into %sequence_num. Note >>>> that >>>> %sequence_num appears twice in the 'lcall' action, since it acts both as an >>>> input parameter for the Perl function and as the variable where the return >>>> value from the function is stored. >>>> There is also an alternative way for addressing the same assignment. This >>>> approach involves keeping the counter for each call ID not in a SEC >>>> context, >>>> but rather in a Perl hash table. That approach is more efficient, since the >>>> counter value is not copied back and forth between the Perl function and >>>> the >>>> SEC context each time it needs to be incremented. Instead, the counter >>>> value >>>> stays in a Perl data structure during these increments. Here is a relevant >>>> rule >>>> example: >>>> type=single >>>> ptype=regexp >>>> pattern=call_id (\S+) >>>> desc=call id $1 >>>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \ >>>> write - the new value for counter for call $1 is %sequence_num >>>> The above example involves the use of a Perl hash (associative array) >>>> 'calls', >>>> where the call ID serves as a key into the hash. For each key (call ID), >>>> the >>>> hash stores the counter for that call ID. Each time the function >>>> sub { ++$calls{$_[0]} } >>>> gets called, the counter for the relevant call ID (the value of $1) is >>>> incremented in the hash 'calls', and the result is stored to %sequence_num. >>>> If you just want to retrieve the current value of the counter for some >>>> call ID >>>> (the value of $1) without incrementing it, you can simply invoke the >>>> following >>>> action: >>>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } ) >>>> With both the context-based and Perl-hash-based approach, one needs to >>>> think >>>> about dropping stale contexts (or stale hash keys) for calls that are no >>>> longer >>>> relevant. With contexts, you can remove them with the 'delete' action. In >>>> the >>>> case you use the Perl hash for storing the counters, you can use the >>>> following >>>> 'lcall' action for deleting a key from the hash 'calls': >>>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } ) >>>> I hope the above examples are useful. >>>> kind regards, >>>> risto >>>> Kontakt Jim Van Meggelen (< [ mailto:jim.vanmegge...@clearlycore.com | >>>> jim.vanmegge...@clearlycore.com ] >) kirjutas kuupäeval R, 22. september >>>> 2023 >>>> kell 19:50: >>>>> I am using SEC for an atypical use case, where rather than using log >>>>> events to >>>>> correlate events for things like security events, I am using it to parse >>>>> through Asterisk logs in order to capture a sequence of phone call events >>>>> (call >>>>> comes in, auto attendant answers, user selects a digit, call is >>>>> transferred, >>>>> etc). I have SEC write each to a CSV file, which can then be processed >>>>> downstream for reporting (for example, I produced a Sankey chart using >>>>> this >>>>> data). >>>>> SEC has proven fantastic for this, however one minor issue has been that >>>>> the >>>>> timestamps in the logs are not granular to smaller than a second, so it's >>>>> possible for two or more events to occur within the same second. >>>>> Generally this >>>>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can >>>>> result in out-of-sequence lines, if they both contain the exact same >>>>> timestamp. >>>>> So, what I've been trying to figure out is how to store a counter that is >>>>> tied >>>>> to the uniqueid of the call, and increment that with each event. I >>>>> figured I'd >>>>> be able to do this by storing an integer in the event store of a context >>>>> (tied >>>>> to the uniqueid for that call). I can then increment it as the various log >>>>> lines of the call are processed in turn.
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
