Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
ok, if you are losing packets over loopback to the oms agent, that's a problem with it not with rsyslog. David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote: I hadn't even noticed that... OK, time to go speak with someone else... Port 25224 (UDP) is the OMS Agent's Syslog. Port 25226

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
I hadn't even noticed that... OK, time to go speak with someone else... Port 25224 (UDP) is the OMS Agent's Syslog. Port 25226 (TCP) is for CEF events. With the events being dropped at a different application (OMS Agent for Linux), this isn't something you can likely assist with. I'll submit a

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
ahh, I thought that was a rsyslog thread that could be maxing out a core. my logging strategy is that everything should get sent to the central syslog server, and only there should it get thrown away (and before you throw it away, consider counting it, the number of times that an uninteresting

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
I think my best course right now is to tie down and remove what I can from the Syslog Collector side of this. At minimum it'll reduce the amount of work rsyslog has to do. If I can completely remove the regex and contains searches in favour of syslog tags/properties, we'll all be better off and

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
As I said before, log some of the messages with the template RSYSLOG_DebugFormat and see what you have and how you can filter more efficiently. Rsyslog is very efficient at processing messages, but regex and contains are about the most expensive tests that you can do If you really do need

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
using the new action() syntax, you can name the actions so they aren't just numbered. starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
the fact that action2 is failingh frequently needs to be addressed, that will cause you all sorts of grief On Tue, 15 Nov 2022, Redbourne,Michael wrote: Date: Tue, 15 Nov 2022 20:01:52 + From: "Redbourne,Michael" To: David Lang Cc: rsyslog-users Subject: RE: [rsyslog] rsyslog

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0: origin=core.action processed=1628 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
you have the impstats module loaded in your config and writing stats out, please post the output of this. David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote: Date: Tue, 15 Nov 2022 19:38:27 + From: "Redbourne,Michael" To: David Lang Cc: rsyslog-users Subject: RE: [rsyslog]

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
I'm still not understanding what you mean by pstats - it's not a package or command available to me. It's apart of Unix from what I can tell. I've placed below the unparsed information form /proc/net/netstat and /proc/net/udp /proc/net/netstat TcpExt: SyncookiesSent SyncookiesRecv

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
what does the pstats output look like when it's dropping messages? (give a couple cycles please) did you try to eliminate the action queue for /var/log/secure? David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote: Date: Tue, 15 Nov 2022 13:01:02 + From: "Redbourne,Michael" To:

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
On Tue, 15 Nov 2022, Redbourne,Michael wrote: Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs: Udp: 5820820 packets received 1504 packets to unknown port received. 798900 packet receive errors 3338814

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
Building on this - When the drop count spikes top is showing a spike in CPU usage among the previously listed threads: In:imdup spikes to ~10% in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25% usage. -Original Message- From: rsyslog On Behalf Of Redbourne,Michael via

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Redbourne,Michael via rsyslog
Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs: Udp: 5820820 packets received 1504 packets to unknown port received. 798900 packet receive errors 3338814 packets sent 798900 receive buffer errors

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Rainer Gerhards via rsyslog
Just wanted to make sure awareness of that option. Agree that it is not often needed. Rainer El mar, 15 nov 2022 a las 10:02, David Lang () escribió: > > I haven't needed to do that to handle 300k messages/sec on UDP input (usually > I > run into bottlenecks in processing the messages long

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread David Lang via rsyslog
I haven't needed to do that to handle 300k messages/sec on UDP input (usually I run into bottlenecks in processing the messages long before I have problems accepting them) David Lang On Tue, 15 Nov 2022, Rainer Gerhards wrote: let me add: look into setting imudp to realtime priority. Doc:

Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events

2022-11-15 Thread Rainer Gerhards via rsyslog
let me add: look into setting imudp to realtime priority. Doc: https://www.rsyslog.com/doc/master/configuration/modules/imudp.html Rainer El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog () escribió: > > Some additional comments on the config > > > > These action queue configs probably