Date: Thu, 12 Sep 2019 12:59:38 +0200
From: Lucien Courcol <[email protected]>
To: David Lang <[email protected]>
Cc: Lucien Courcol via rsyslog <[email protected]>
Subject: Re: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
~10min after starting
Hello,
So I loaded the impstats.so module and restarted Rsyslog.
Rsyslog stopped forwarding logs in less than 5 min after this first restart
so I didn't get any rsyslogd-pstats logs (I use the default option to send
stats to the syslog stream with interval set to 5min).
I restarted Rsyslog and this time it kept working a bit longer. After 1 hr
Rsyslog stopped forwarding logs.
Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this:
2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0 discarded.nf=0
maxqsize=145 *(last log)*
2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q:
origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0 discarded.nf=0
maxqsize=145
Another filter in Splunk ( "rsyslogd-pstats" submitted NOT "origin=imptcp"
NOT "origin=imtcp" ) gave me this:
2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) *
2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=91 disallowed=0
2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=6009 ratelimit.discarded=0
ratelimit.numratelimiters=0
2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0
2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=83 disallowed=0
2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=5139 ratelimit.discarded=0
ratelimit.numratelimiters=0
2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=0 disallowed=0
2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - -
imudp(*:514): origin=imudp submitted=66 disallowed=0
2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock:
origin=imuxsock submitted=4657 ratelimit.discarded=0
ratelimit.numratelimiters=0
* I have the same issue with the another vCenter 6.7U3. *
Rsyslog stops forwarding logs after a while.
Both have the same Rsyslog version:
rsyslogd 8.37.0, compiled with:
PLATFORM: x86_64-unknown-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Number of Bits in RainerScript integers: 64
This is my /etc/rsyslog.conf (identical on both vCenter):
################################################################################
############################# VMware Rsyslog Configuration
####################
################################################################################
###### Module declarations ######
$ModLoad impstats.so
$ModLoad imuxsock.so
$ModLoad imtcp.so # TLS
$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100
ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5%
full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 1514
#
# cron log entries for GEN003160
#
cron.* -/var/log/cron
#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log
*However*, in a 3rd vCenter, which is running a slightly different version
(VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue.
This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine:
################################################################################
############################# VMware Rsyslog Configuration
####################
################################################################################
###### Module declarations ######
$ModLoad imuxsock.so
$ModLoad imtcp.so # TLS
$ModLoad imptcp.so # TCP
$ModLoad imudp.so # UDP
$ModLoad omrelp.so # RELP
###### Common configuration ######
$EscapeControlCharactersOnReceive off
###### Template declarations ######
$template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template vpxdFmt,"%msg%\n"
$template rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log"
$template rsyslogadminFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%app-name% %msg%\n"
$template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log"
$template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text%
%hostname% %app-name% %msg%\n"
$template defaultSystemLoc,"/var/log/vmware/messages"
###### Rule declarations ######
# TCP/UDP/rsyslog input ruleset declaration
$RuleSet all
# Make gtls driver the default
$DefaultNetstreamDriver gtls
# Shared certificate authority certificate
$DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt
# Client certificate
$DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt
# Client key
$DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key
# Include the configuration for syslog relay
# _must_ be first to relay all messages
$IncludeConfig /etc/vmware-syslog/syslog.conf
# vmware services
:programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdird" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmafdd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcad" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "rbd" ?defaultLoc;defaultFmt
& stop
:app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt
& stop
:programname, isequal, "vmon" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "vmcamd" ?defaultLoc;defaultFmt
& stop
:programname, isequal, "pod" stop
:programname, isequal, "updatemgr" stop
# vpxd-svcs logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd-svcs" stop
# vmware-hvc logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "hvc" stop
# vpxd logs to its local logs, hence avoiding duplicate logging.
:programname, isequal, "vpxd" stop
# For local host's syslog and system logs use the following rules
# localhost
if $fromhost contains $$myhostname then ?defaultSystemLoc
& stop
#localhost
:fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc
& stop
# ESX rules
# Define large LinkedList action queue with 2K msgs cap to accomodate 100
ESXs
$ActionQueueSize 2000
# Do not choke ESXs, rather start dropping messages after queue is 97.5%
full
$ActionQueueDiscardMark 1950
$ActionQueueDiscardSeverity 0
$ActionQueueTimeoutEnqueue 1
# VC syslog server log collection
*.* ?esxLoc;esxFmt
###### Input server declarations ######
# Setup input flow
$DefaultRuleset all
$InputPTCPServerBindRuleset all
$InputPTCPServerRun 514
$InputUDPServerBindRuleset all
$UDPServerRun 514
$InputTCPServerBindRuleset all
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 1514
The difference is at the end:
The conf file *with the issue* has these lines:
#
# cron log entries for GEN003160
#
cron.* -/var/log/cron
#
# auth.log entries for GEN003660
#
auth.* -/var/log/auth.log
I don't really know what that means and if it can explain the issue.
Lucien
On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol <[email protected]>
wrote:
Thx David Lang,
I'll work on that tomorrow.
I'll use this guide to get impstats logs:
https://www.rsyslog.com/how-to-use-impstats/
Lucien
On Wed, Sep 11, 2019 at 7:56 PM David Lang <[email protected]> wrote:
wihtout seeing your rsyslog.conf we can't begin to guess what's
happening.
configuring impstats and showing us the output when it's stopped would
help us
to see what's happening.
David Lang
On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote:
Date: Wed, 11 Sep 2019 14:45:38 +0200
From: Lucien Courcol via rsyslog <[email protected]>
To: [email protected]
Cc: Lucien Courcol <[email protected]>
Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working
~10min
after starting
Hello,
We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago
and
we noticed a gap of logs in our syslog server (kiwi) since then.
I did a bit of troubleshooting but Rsyslog (the syslog client running on
VCSA) is completely new to me.
I use this command to restart Rsyslog:
systemctl restart rsyslog
Right after starting up Rsyslog, logs are being sent to our syslog
server.
~10min later, no more logs are sent.
The vCenter log file in our syslog server stops getting updated.
I did a tcpdump in our vCenter and I see that the vCenter stops sending
logs.
Using UDP or TCP doesn't fix the issue.
I looked for errors in various log files in the vCenter but can't find
anything.
This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like
after
restarting Rsyslog
2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin
software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="
http://www.rsyslog.com";] exiting on signal 15.
2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable
TZ
is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
http://www.rsyslog.com/e/2442 ]
2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX
socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin
software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
http://www.rsyslog.com";] start
Rsyslog is still running based on this command
systemctl status rsyslog.service
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
vendor
preset: enabled)
Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 22235 (rsyslogd)
Tasks: 12
Memory: 5.7M
CPU: 191ms
CGroup: /system.slice/rsyslog.service
└─22235 /usr/sbin/rsyslogd -n
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main
process exited, code=killed, status=9/KILL
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging
Service.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit
entered failed state.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed
with result 'signal'.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging
Service...
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging
Service.
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment
variable
TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try
http://www.rsyslog.com/e/2442 ]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired
UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.
[v8.37.0]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin
software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="
http://www.rsyslog.com";] start
(real hostname has been replaced by vcenter.domain.local)
I created a ticket at VMware support, but the agent wasn't able to find
any
errors as well and she suggested to take a backup of our vCenter and
reinstall with a restore to get a fresh install of Photon OS since
Rsyslog
is integrated in Photon OS. I'm not going to do that now, maybe as a
last
troubleshooting step.
In the meantime, do you guys have an idea? Wrong Rsyslog config?
Thx for your help.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
