ive ~ $.02
From: rsyslog on behalf of Rainer Gerhards
via rsyslog
Sent: Wednesday, November 16, 2022 3:14 AM
To: rsyslog-users
Cc: Rainer Gerhards
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> @rainer, what is the in_syslog.rb thread that he saw maxin
ate: Tue, 15 Nov 2022 21:39:45 +
> > From: "Redbourne,Michael"
> > To: David Lang
> > Cc: rsyslog-users
> > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
> >
> > I'm going to reach out to networking folks and see if
vid Lang ; rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
ahh, I thought that was a rsyslog thread that could be maxing out a core.
my logging strategy is that everything should get sent to the central syslog
server, and only there should it get thrown a
ant 127.0.0.1.
-Original Message-
From: David Lang
Sent: Tuesday, November 15, 2022 6:06 PM
To: Redbourne,Michael
Cc: David Lang ; rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
ahh, I thought that was a rsyslog thread that could be maxing out a c
decimal)
David Lang
On Tue, 15 Nov 2022, Redbourne,Michael wrote:
Date: Tue, 15 Nov 2022 21:55:48 +
From: "Redbourne,Michael"
To: David Lang
Cc: rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
I think my best course right now is t
Tuesday, November 15, 2022 5:49 PM
To: Redbourne,Michael
Cc: David Lang ; rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
As I said before, log some of the messages with the template
RSYSLOG_DebugFormat and see what you have and how you can filter more
eff
slog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
I'm going to reach out to networking folks and see if I can get something better in
place, especially around negating logs further up the chain then the syslog collector.
(Moreso related to the Checkpoint
rom: David Lang
Sent: Tuesday, November 15, 2022 4:19 PM
To: Redbourne,Michael
Cc: David Lang ; rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
using the new action() syntax, you can name the actions so they aren't just
numbered.
starting rsyslog with -o /pa
, Redbourne,Michael wrote:
Date: Tue, 15 Nov 2022 20:01:52 +
From: "Redbourne,Michael"
To: David Lang
Cc: rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Tue Nov 15 00:22:18 2022: global: origin=dynstats
Tue Nov 15 00:22:18 2022: imuxsock: origi
2022 3:43 PM
To: Redbourne,Michael
Cc: David Lang ; rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
you have the impstats module loaded in your config and writing stats out,
please post the output of this.
David Lang
On Tue, 15 Nov 2022, Redbourne,Mic
=51446
-Original Message-
From: David Lang
Sent: Tuesday, November 15, 2022 3:43 PM
To: Redbourne,Michael
Cc: David Lang ; rsyslog-users
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
you have the impstats module loaded in your config and writing stats out,
please
: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
I'm still not understanding what you mean by pstats - it's not a package or
command available to me. It's apart of Unix from what I can tell. I've placed
below the unparsed information form /proc/net/netstat and /proc/net/udp
/proc/net/nets
ect: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
what does the pstats output look like when it's dropping messages? (give a
couple cycles please)
did you try to eliminate the action queue for /var/log/secure?
David Lang
On Tue, 15 Nov 2022, Redbourne,Michael wrote:
> Dat
chael"
To: rsyslog-users , David Lang
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Building on this -
When the drop count spikes top is showing a spike in CPU usage among the
previously listed threads:
In:imdup spikes to ~10%
in_syslog.rb spikes to 90-100% usage
rs:main Q:
you are doing so via
'contains'
David Lang
-Original Message-
From: rsyslog On Behalf Of Rainer Gerhards
via rsyslog
Sent: Tuesday, November 15, 2022 5:11 AM
To: David Lang
Cc: Rainer Gerhards ; rsyslog-users
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
rsyslog
Sent: Tuesday, November 15, 2022 8:42 AM
To: rsyslog-users ; David Lang
Cc: Redbourne,Michael
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Concerning the /proc and pstats. There is /proc/net/netstat, which looks
something like this after a couple minutes of logs
Cc: Rainer Gerhards ; rsyslog-users
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Just wanted to make sure awareness of that option. Agree that it is not often
needed.
Rainer
El mar, 15 nov 2022 a las 10:02, David Lang () escribió:
>
> I haven't needed to do that t
Just wanted to make sure awareness of that option. Agree that it is
not often needed.
Rainer
El mar, 15 nov 2022 a las 10:02, David Lang () escribió:
>
> I haven't needed to do that to handle 300k messages/sec on UDP input (usually
> I
> run into bottlenecks in processing the messages long
I haven't needed to do that to handle 300k messages/sec on UDP input (usually I
run into bottlenecks in processing the messages long before I have problems
accepting them)
David Lang
On Tue, 15 Nov 2022, Rainer Gerhards wrote:
let me add: look into setting imudp to realtime priority. Doc:
let me add: look into setting imudp to realtime priority. Doc:
https://www.rsyslog.com/doc/master/configuration/modules/imudp.html
Rainer
El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
() escribió:
>
> Some additional comments on the config
>
>
>
> These action queue configs probably
Some additional comments on the config
These action queue configs probably don't do what you intend them to do
the first thing is that they only affect the next action, which is authpriv.* to
/var/log/secure and you configure 2000 threads to write these logs out. That
will create a HUGE
Lang
Michael Redbourne
-Original Message-
From: David Lang
Sent: Monday, November 14, 2022 9:52 AM
To: Redbourne,Michael via rsyslog
Cc: Redbourne,Michael
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
16 threads for UDP receive is very incorrect. Rsyslo
David Lang
Sent: Monday, November 14, 2022 9:52 AM
To: Redbourne,Michael via rsyslog
Cc: Redbourne,Michael
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
16 threads for UDP receive is very incorrect. Rsyslog should only need a single
thread, even when receiving m
16 threads for UDP receive is very incorrect. Rsyslog should only need a single
thread, even when receiving messages at a rate of hundreds of thousands of
messages/sec. too many threads will slow rsyslog down and it will use
recvmmesg() to pull multiple udp messages from the OS buffers in a
Hey folks,
Hoping someone has the expertise to help me out here. We have a Syslog server
running CentOS 7.9, kernel 5.18.5. It's acting as the centralized point for
Syslog (TCP + UDP) ingestion for 100+ Syslog devices. Something is causing a
what I think is the kernel (udp_queue_rcv_one_skb)
25 matches
Mail list logo
