hi Jim, many thanks for the clarification! I have a better understanding now about the nature of the problem. Because you are doing offline processing of an already existing log file, removing old contexts or old hash table keys (recommendation from the end of my post) might not be so important, provided that call IDs are not reused for different calls in the processed log file, and the number of call IDs is not very large. That would keep the solution a bit more simple. kind regards, risto
Kontakt Jim Van Meggelen (<jim.vanmegge...@clearlycore.com>) kirjutas kuupäeval L, 23. september 2023 kell 22:11: > Risto, > > I cannot change the timestamps, as they are coded in the files I'm working > with (old data). Going forward, I also cannot guarantee that I will have > control over the source data, and it is likely that it will have a similar > granularity. Finally, there's no need for that level of granularity > downstream; it is only necessary that two events with the same timestamp > are sequenced so that it is clear which happened first. > > It does seem a bit kludgy, but if I try to reduce the kludge in my SEC > file, I am almost certain to have to introduce kludge somewhere else, which > is worse (this way I only have SEC to maintain, rather than additional > technology). > > This way, I can feed relevant log files to SEC from any source, and trust > that I'll be able to handle them, so long as they follow standard Asterisk > log format. > > Thanks again for your generous advice. > > Jim > > > > -- > Jim Van Meggelen > ClearlyCore Inc. > > > > +1-416-639-6001 (DID) > +1-877-253-2716 (Canada) > +1-866-644-7729 (USA) > +1-416-425-6111 x6001 > jim.vanmegge...@clearlycore.com > http://www.clearlycore.com > > > *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:* > https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf > > ------------------------------ > > *From: *"Risto Vaarandi" <risto.vaara...@gmail.com> > *To: *"Jim Van Meggelen" <jim.vanmegge...@clearlycore.com> > *Cc: *"simple-evcorr-users" <simple-evcorr-users@lists.sourceforge.net> > *Sent: *Saturday, 23 September, 2023 03:51:02 > *Subject: *Re: [Simple-evcorr-users] Storing a sequence counter in a > context > > hi Jim, > the solutions from my previous post were provided without knowing the root > cause of the issue. If the main reason for the problem is the nature of the > timestamps in the logs (that is, they are provided with an accuracy of a > second), I would recommend using a different logging scheme with > high-resolution timestamps. For example, as David has mentioned in his > post, using syslog servers with RFC5424-based syslog protocol ( > https://datatracker.ietf.org/doc/html/rfc5424) provides such timestamps > out-of-the-box. I am not sure whether you can change the logging scheme for > these particular application messages, but if you can, these timestamps are > highly useful for many purposes like the investigation of past system > faults and security incidents. > kind regards, > risto > > > hi Jim, >> >> let me provide some suggestions how to accomplish this task. First, you >> could utilize the context based approach that you have described in your >> post. In that case, the numeral that has been retrieved from the context >> with the 'pop' action needs to be incremented, and you can do it with the >> 'lcall' action that invokes the relevant Perl code. Here is one example (I >> have added several of extra actions into the rule for initializing the >> context): >> >> type=single >> ptype=regexp >> pattern=call_id (\S+) >> desc=call id $1 >> action=exists %e call_uniqueid_seq_$1; \ >> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \ >> pop call_uniqueid_seq_$1 %sequence_num; \ >> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \ >> add call_uniqueid_seq_$1 %sequence_num >> >> The first two actions check if the context call_uniqueid_seq_* for the >> given call exists, and if not, 0 is written into that context to initialize >> it: >> >> exists %e call_uniqueid_seq_$1; \ >> if %e ( none ) else ( add call_uniqueid_seq_$1 0); >> >> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) >> are identical to the actions from your email. >> >> The 'lcall' action (4th action in the above rule) increments the counter >> value retrieved from the context by executing the following Perl one-line >> function: >> sub { $_[0] + 1 } >> That function takes the value of its parameter (that is, %sequence_num) >> and returns an incremented value which is stored back into %sequence_num. >> Note that %sequence_num appears twice in the 'lcall' action, since it acts >> both as an input parameter for the Perl function and as the variable where >> the return value from the function is stored. >> >> There is also an alternative way for addressing the same assignment. This >> approach involves keeping the counter for each call ID not in a SEC >> context, but rather in a Perl hash table. That approach is more efficient, >> since the counter value is not copied back and forth between the Perl >> function and the SEC context each time it needs to be incremented. Instead, >> the counter value stays in a Perl data structure during these increments. >> Here is a relevant rule example: >> >> type=single >> ptype=regexp >> pattern=call_id (\S+) >> desc=call id $1 >> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \ >> write - the new value for counter for call $1 is %sequence_num >> >> The above example involves the use of a Perl hash (associative array) >> 'calls', where the call ID serves as a key into the hash. For each key >> (call ID), the hash stores the counter for that call ID. Each time the >> function >> sub { ++$calls{$_[0]} } >> gets called, the counter for the relevant call ID (the value of $1) is >> incremented in the hash 'calls', and the result is stored to %sequence_num. >> >> If you just want to retrieve the current value of the counter for some >> call ID (the value of $1) without incrementing it, you can simply invoke >> the following action: >> >> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } ) >> >> With both the context-based and Perl-hash-based approach, one needs to >> think about dropping stale contexts (or stale hash keys) for calls that are >> no longer relevant. With contexts, you can remove them with the 'delete' >> action. In the case you use the Perl hash for storing the counters, you can >> use the following 'lcall' action for deleting a key from the hash 'calls': >> >> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } ) >> >> I hope the above examples are useful. >> >> kind regards, >> risto >> >> >> >> >> >> >> >> >> Kontakt Jim Van Meggelen (<jim.vanmegge...@clearlycore.com>) kirjutas >> kuupäeval R, 22. september 2023 kell 19:50: >> >>> I am using SEC for an atypical use case, where rather than using log >>> events to correlate events for things like security events, I am using it >>> to parse through Asterisk logs in order to capture a sequence of phone call >>> events (call comes in, auto attendant answers, user selects a digit, call >>> is transferred, etc). I have SEC write each to a CSV file, which can then >>> be processed downstream for reporting (for example, I produced a Sankey >>> chart using this data). >>> >>> SEC has proven fantastic for this, however one minor issue has been that >>> the timestamps in the logs are not granular to smaller than a second, so >>> it's possible for two or more events to occur within the same second. >>> Generally this doesn't cause a problem, but when sorting the CSV file >>> elsewhere, this can result in out-of-sequence lines, if they both contain >>> the exact same timestamp. >>> >>> So, what I've been trying to figure out is how to store a counter that >>> is tied to the uniqueid of the call, and increment that with each event. I >>> figured I'd be able to do this by storing an integer in the event store of >>> a context (tied to the uniqueid for that call). I can then increment it as >>> the various log lines of the call are processed in turn. >>> >>> >>> >
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
