Hi Dave,
would it be possible for you to attach to the running rsyslogd process
using gdb to see which thread is looping?
gdb attach [PID]
best regards,
Andre Lorbach
> -----Original Message-----
> From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of
Dave
> Cottlehuber
> Sent: Thursday, September 21, 2017 1:49 PM
> To: rsyslog@lists.adiscon.com
> Subject: [rsyslog] 100% cpu lockup rsyslog 8.29 on FreeBSD 11.0Rp12
amd64 -
> bad config?
>
> hi folks,
>
> for the last few weeks I have been seeing rsyslog locking up a single
cpu at
> 100% on multiple servers. All are running FreeBSD 11.0R amd64p12, and
> rsyslog 8.29, although the lockup has been seen with prior patch levels
and
> rsyslog versions.
>
> The general architecture is as follows:
>
> - rsyslog replaces normal freebsd syslog
> - listens on UDP & TCP both on jails and on main host
> - does basic template tidyups to correct hostnames
> - forwards these over a spiped[1] secure connection rather than inbuilt
TLS to
> a centralised rsyslog logging host
> - central rsyslog sticks these in a file and also throws them into
graylog[2]
> - the local rsyslog is supposed to be resilient to reboots of the
central system,
> and to buffer locally via queue any messages in case of e.g. reboots or
> transient network issues, but I don't believe this is working as
expected
>
> Is it possible my config[3] is somehow incorrect, and somehow causing
this
> lock up?
>
> [1]: http://www.tarsnap.com/spiped.html
> [2]: https://graylog.org/
> [3]:
> https://gist.github.com/dch/c5dcb7b6159dc2b070aefe89207c3a92#file-
> rsyslog-conf
>
>
> # /usr/local/etc/rsyslog.conf
> # central logging server is largely the same except has an additional
stanza for
> graylog # Load Modules
> module(load="imtcp")
> module(load="imklog")
> module(load="imudp")
> module(load="imrelp")
> module(load="omrelp")
> module(load="imuxsock")
> module(load="immark")
> module(load="impstats")
> module(load="imfile" PollingInterval="5")
>
> # Disk Queues
> # zfs create -o canmount=off zroot/var/spool # zfs create
> zroot/var/spool/rsyslog # mkdir -m0700 /var/log/rsyslog $WorkDirectory
> /var/spool/rsyslog $MaxMessageSize 64k $ActionQueueFileName mainq
> $ActionQueueType LinkedList $ActionQueueSaveOnShutDown on
> $ActionQueueMaxDiskSpace 10g $ActionSendResendLastMsgOnReconnect
> on $ActionResumeRetryCount -1
>
> # FreeBSD kernel / console logging
> $KLogPermitNonKernelFacility on
> $KlogConsoleLogLevel 10
>
> # rsyslog templates
>
> ## handle multiline messages (e.g. from various APIs safely)
> $EscapeControlCharactersOnReceive="off"
>
> ## ensure hostnames make sense when coming via UDP / jails
> template(name="normalised" type="string"
> string="<%pri%>%protocol-version% %timestamp:::date-rfc3339%
> example.org
> %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")
> # rsyslog input modules
> input(type="imtcp"
> address="127.0.0.1"
> port="514")
> input(type="imudp"
> rcvbufSize="1m"
> address="127.0.0.1"
> port="514")
> # allow logging via jail hosts
> input(type="imtcp"
> address="10.241.0.0"
> port="514")
> input(type="imudp"
> rcvbufSize="1m"
> address="10.241.0.0"
> port="514")
>
> # filters
>
> :msg, contains, "NSSWITCH(_nsdispatch)" ~
>
> # rulesets
>
> # rsyslog output modules
>
> # base servers forward logs using the hostname-fixing template
> action(type="omrelp"
> target="127.0.0.1"
> template="normalised"
> port="14514"
> )
>
> # log locally just like normal syslog with hostnames adjusted
> action(type="omfile"
> queue.type="linkedlist"
> template="normalised"
> file="/var/log/messages"
> )
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond
> our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
