Thank you. "action.execOnlyOnceEveryInterval" is very useful.
We intend to keep all unfiltered syslogs in a local file, say file1,
but log a rate limited message to file2 and forward that message to a
syslog collector.
Is it possible to change the facility, severity or message for the next actions?
Thanks again.
On Sun, Apr 7, 2024 at 11:42 PM David Lang <da...@lang.hm> wrote:
>
> you would need to do the filtering with omprog on the sending machines.
>
> I would suggest that rather than throwing them away, you generate a log every
> rate-limiting period along the lines of "X number of messages happened in the
> last period"
>
> This is really a job for an event correlation engine, since rsyslog is
> multi-threaded for performance, it can't count on anything internal seeing
> every
> message, which is why filtering for the messages and sending them to an
> external
> app (single threaded or a very complex multi-threaded thing that correlates
> across it's threads) is the right thing to do.
>
> David Lang
>
> On Sun, 7 Apr 2024, Prasad Koya via rsyslog wrote:
>
> > Thanks for the replies.
> >
> > My requirement is that I have a daemon that may generate a burst of
> > syslogs, say, every minute (when a certain subsystem is overloaded).
> > We do not want to write all these syslogs to /var/log/messages. We
> > also do not want to forward so many of those syslogs to a remote
> > syslog server. We wanted to rate limit them, maybe 1 per hour or so.
> > For now, I have tried 'omprog' module and sending such syslogs to an
> > external program.
> >
> >
> > module( load="omprog" )
> >
> > if ($syslogfacility-text == 'daemon' and $msg contains "Out of memory")
> > then {
> > action(type="omprog"
> > binary="/usr/bin/rate_limit_daemon_syslogs.sh"
> > template="RSYSLOG_FileFormat")
> > stop
> > }
> >
> > /usr/bin/rate_limit_daemon_syslogs.sh maintains state on when was the
> > last syslog forwarded or written locally. If this script decides its
> > ok to syslog, then it will call 'logger' with differently worded
> > message and different facility.
> >
> > In our case we'd like to limit these logs at the source and not at an
> > intermediate syslog collector. I took a quick look at the sampling
> > docs and that may not apply to our scenario.
> >
> > Thanks.
> >
> > On Wed, Apr 3, 2024 at 12:07 AM Mariusz Kruk via rsyslog
> > <rsyslog@lists.adiscon.com> wrote:
> >>
> >> A piece of feedback against
> >> https://www.rsyslog.com/doc/tutorials/random_sampling.html
> >>
> >> I know it's obvious for some people but for some it might not be - it
> >> will collect _about_ 20% of logs provided the overall number of events
> >> is big enough. Since it's a random-based mechanism, it's... random.
> >>
> >> Additionally, I had a similar thing implemented for load-balancing to
> >> different "backends" and for some reason it didn't work uniformly (I
> >> think I already wrote about this) - one of four backends was never
> >> chosen. The thing is I was choosing not based directly on random(4) but
> >> on (random(some_bigger_integer) % 4) which I thought would yield more
> >> uniform randomness. It appears I was wrong.
> >>
> >> MK
> >>
> >>
> >> On 3.04.2024 08:51, Rainer Gerhards via rsyslog wrote:
> >> > This sounds a bit like you are looking for this:
> >> >
> >> > https://www.rsyslog.com/doc/tutorials/log_sampling.html
> >> >
> >> > HTH
> >> > Rainer
> >> >
> >> > El mié, 3 abr 2024 a las 3:25, Prasad Koya via rsyslog
> >> > (<rsyslog@lists.adiscon.com>) escribió:
> >> >> Hi
> >> >>
> >> >>
> >> >> module( load="imuxsock" )
> >> >> module( load="imklog" )
> >> >>
> >> >> if ($syslogfacility-text == 'kern' and $msg contains "Out of memory")
> >> >> then {
> >> >> action(type="omfile" queue.size="1000" queue.type="LinkedList"
> >> >> queue.dequeueSlowDown="3600000000" file="/var/log/oom")
> >> >> }
> >> >>
> >> >>
> >> >> If we have a flurry of, for example, "Out of memory" messages from the
> >> >> facility 'kern', we'd like to keep only, say, one per hour in a
> >> >> separate file.
> >> >>
> >> >> https://rsyslog.readthedocs.io/en/latest/rainerscript/queue_parameters.html
> >> >> says do not set low value for queue.size.
> >> >>
> >> >> Above page also says queue.dequeueslowdown can be used as
> >> >> rate-limiting. Maybe that doesn't apply for "omfile" as I don't see
> >> >> above rsyslog.conf working. high and low watermarks are not applicable
> >> >> here as it's not a disk based queue. Appreciate any pointers.
> >> >>
> >> >> Thank you.
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> http://www.rsyslog.com/professional-services/
> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> >> >> if you DON'T LIKE THAT.
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> > DON'T LIKE THAT.
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
