Bug#367526: swatch: threshold not actually implemented as per Debian docs

[Nikolay Kokalichev]

Nick Leverton <[EMAIL PROTECTED]> wrote:

Package: swatch
Version: 3.1.1-2
Severity: normal

The revised docs on throttle/threshold from bug 292584 don't match the upstream code
- but regrettably the upstream re-implementation of threshold doesn't work well anyway.
 I agree that the implementation of the threshold is not working well.

Like you, I believe the syntax the upstream code is trying to implement
is as follows, so that 'threshold' is just an integer limit that the
throttle must meet before it triggers:

throttle hh:mm:ss,threshold=n,repeat=(yes|no),key=(message|regex|)

with 'n' as a simple count (no longer contains a separate time factor) - this is
clear from the code;
and presumably with hh:mm:ss being the time for the threshold, as usual
for throttle.

However the upstream code seems incorrect. Firstly it never applies
the time limit if there is a threshold. Second, the code ignores the
repeat= option and always behaves as repeat=no; and thirdly, the first
message matching after restarting swatch is always acted on regardless
of the threshold.


I've attached a patch which corrects the code to match the amended Debian
doc and fixes the above problems, and further alters the amended docs to
fully match the fixed code.

I hope you agree with my interpretation of what upstream intended, if
not please feel free to re-work it, and thank you for your efforts,

I'll make some tests on your patch and if it's working well will correct the package to match your threshold interpretation.

To be short I would like to tell that the source code of swatch is not written well also there is no error checking at all and it need it be fully revised.

Nick Leverton

Thank you for  the patch and for the time you spent to wrote it.


-- System Information:
Debian Release: 3.1
APT prefers testing
APT policy: (60, 'testing'), (2, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.15-1-k7
Locale: LANG=en_GB.ISO-8859-15, LC_CTYPE=en_GB.ISO-8859-15 (charmap=ISO-8859-15)
--- lib/Swatch/Throttle.pm.orig 2004-07-19 21:14:54.000000000 +0100
+++ lib/Swatch/Throttle.pm 2006-05-16 18:12:50.000000000 +0100
@@ -98,7 +98,15 @@
my @dmyhms;
my $key;
my $cur_rec;
- my $msg = $opts{"MESSAGE"};
+ my $ret = "";
+
+ my $threshold = (exists $opts{THRESHOLD} ? $opts{THRESHOLD} : 1);
+ my $repeat;
+ if (exists $opts{REPEAT}) {
+ $repeat = ( $opts{REPEAT} =~ /^yes$/i ); # was repeat= specified ? if so remember it.
+ } else {
+ $repeat = ( exists $opts{THRESHOLD} ); # default=yes if we have a threshold, else no
+ }

## get the time ##
if ($opts{TIME_FROM} eq 'realtime') {
@@ -128,37 +136,46 @@
}

## just make the record if it doesn't exist yet ##
- if (not defined $LogRecords{$key}) {
- my $rec = ();
- $rec->{KEY} = $key;
- $rec->{FIRST} = [ @dmyhms ];
- $rec->{LAST} = [ @dmyhms ];
- $rec->{HOLD_DHMS} = $opts{HOLD_DHMS} if defined $opts{HOLD_DHMS};
- $rec->{COUNT} = 1;
- $LogRecords{$key} = $rec;
- return $msg;
- } else {
+ if (defined $LogRecords{$key}) {
$cur_rec = $LogRecords{$key};
$cur_rec->{COUNT}++;
- if (defined $opts{THRESHOLD} and $cur_rec->{COUNT} == $opts{THRESHOLD}) {
- ## threshold exceeded ##
- chomp $msg;
- $msg = "$msg (threshold $opts{THRESHOLD} exceeded)";
- $cur_rec->{COUNT} = 0;
- } elsif (defined $opts{HOLD_DHMS}
- and past_hold_time($cur_rec->{LAST},
- [EMAIL PROTECTED], $opts{HOLD_DHMS})) {
+ $cur_rec->{UNPRINTED}++;
+ } else {
+ $cur_rec = ();
+ $cur_rec->{KEY} = $key;
+ $cur_rec->{FIRST} = [ @dmyhms ];
+ $cur_rec->{LAST} = [ @dmyhms ]; # time of start of throttle period
+ $cur_rec->{HOLD_DHMS} = $opts{HOLD_DHMS} if exists $opts{HOLD_DHMS};
+ $cur_rec->{COUNT} = 1; # count seen in this throttle period
+ $cur_rec->{UNPRINTED} = 1; # count seen since last message printed
+ }
+
+ # were we throttling and has the throttle period finished ?
+ if (exists $opts{HOLD_DHMS}
+ && past_hold_time($cur_rec->{LAST}, [EMAIL PROTECTED], $opts{HOLD_DHMS})) {
## hold time exceeded ##
- chomp $msg;
- $msg = "$msg (seen $cur_rec->{COUNT} times)";
+ $cur_rec->{COUNT} = 1;
+ $cur_rec->{LAST} = [ @dmyhms ];
+ }
+
+ # have we reached the threshold for this throttle period ?
+ if ($cur_rec->{COUNT} == $threshold) {
+ ## threshold exceeded, print it out ##
+ $ret = $opts{"MESSAGE"};
+ chomp $ret;
+ # $ret = "$ret (threshold $opts{THRESHOLD} exceeded)" if exists $opts{THRESHOLD};
+ $ret = "$ret (seen $cur_rec->{UNPRINTED} times)" if $cur_rec->{UNPRINTED} > 1;
+ $cur_rec->{UNPRINTED} = 0;
+ # do we want repeated messages after reaching the threshold ?
+ if ($repeat) {
+ # no, start a new throttle period
$cur_rec->{COUNT} = 0;
$cur_rec->{LAST} = [ @dmyhms ];
- } else {
- $msg = '';
}
- $LogRecords{$key} = $cur_rec if exists($LogRecords{$key}); ## save any new values ##
}
- return $msg;
+
+ $LogRecords{$key} = $cur_rec; ## save any new values ##
+ return $ret;
}

################################################################
--- swatch.orig 2006-05-16 13:03:06.000000000 +0100
+++ swatch 2006-05-16 17:18:01.000000000 +0100
@@ -197,36 +197,39 @@

Use B to send matched lines to I.

-=item B]>
+=item B

Use this action to limit the number of times that the matched pattern
has actions performed on it.

-The B option will cause throttling to be based on the regular
-_expression_ instead of the message.
-
-You can also specify a perl compliant regular _expression_ as the value for B,use>. The default is use="^\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s+(.*)" causes the key to be the syslog message without the timestamp. This is most useful when throttling non-syslog created files.
+=over

-=item B
+B option will cause throttling to be based on the regular
+_expression_ instead of the message.

-This action limits the actions on a matched pattern based on the number of
-times it appears in a given time frame. An action like "threshold 4:60"
-will not perform any actions on that pattern unless it appears 4 times
-within any 60 second period (4:60 is the arbitrary default value).
+You can also specify a perl compliant regular _expression_ as the value for
+B. The default is use="^\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s+(.*)"
+causes the key to be the syslog message without the timestamp. This is
+most useful when throttling non-syslog created files.
+
+B further limits the throttling on a matched
+pattern so that it will not be acted on until the specified number of
+events have been matched in the throttle time frame. An action like
+"throttle 0:0:60,threshold=4" will not perform any actions on that
+pattern unless it appears 4 times within 60 seconds.

-The B option will prevent the timer from being reset after the
+B will prevent the timer from being reset after the
threshold has been reached. By default (repeat=yes), once the pattern has
-been triggered, a new 60 second period is begun, which means that if the
-patterns match quickly enough, a threshold of 4:60 could mean that 1 in
-every 4 messages is reported. By using B, you indicate that
-there is not to be more than one match every time interval.
-
-Note that this is similar to, but different from, the standard "throttle"
-command, since "throttle" shows the first line and ignores the rest, while
-"threshold" shows the last line and ignores the preceeding (and optionally
-the following). However, an action like "threshold 1:120" should perform
-similarly to "throttle 0:2:0,use=regex" and has the advantage of not relying
-on a particular timestamp format in the source log entry.
+been triggered, a new period is begun, which means that if the patterns
+match quickly enough, a threshold of 4 events in 60 seconds could mean
+that 1 in every 4 messages is reported. By using B, you
+indicate that there is not to be more than one match every time interval.
+
+Thus "throttle" shows the first line and ignores the rest, while
+"throttle threshold=N" shows line N and ignores the preceeding (and
+optionally the following).
+
+=back

=item B

@@ -242,7 +245,7 @@

=head1 SPECIAL OPTION

-The following may be used as an option for any of the above actions except for throttle and threshold.
+The following may be used as an option for any of the above actions except for throttle.

=over 4

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com