Hello David, Have you received my previous email? Did I configure the impstats module correctly? I've added attachments and I was wondering if it was blocked or moved to your spam folder.
Lucien On Thu, Sep 12, 2019 at 6:00 PM Lucien Courcol <[email protected]> wrote: > Hello David, > > Find attached 2 impstats log files (output to local file, interval of 1 > min.) > Each file comes from a different vCenter with the same issue. > Rsyslog version is the same: 8.37.0 > > rsyslog-stats-vcenter-lab.log => Rsyslog stopped forwarding logs at > 17:26:33 > rsyslog-stats-vcenter-corp.log => Rsyslog stopped forwarding logs > at 17:09:42 > > Lucien > > On Thu, Sep 12, 2019 at 1:08 PM David Lang <[email protected]> wrote: > >> when you configure impstats, you can configure it to write to a local >> file and >> not to syslog, that should decouple it from whatever is causing it to >> freeze.I >> would also set it to log more frequently. >> >> all the options for impstats are listed at >> https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html >> >> each timeperiod it spits out a good chunk of logs, we will want to see at >> least >> one complete set, ideally after it stops processing things otherwise. >> >> putting a queue on a write to disk ( *.* ?esxLoc;esxFmt ) is usually >> counterproductive, rsyslog spends more time managing the queue than just >> writing >> the file. >> >> David Lang >> >> >> On Thu, 12 Sep 2019, Lucien Courcol wrote: >> >> > Date: Thu, 12 Sep 2019 12:59:38 +0200 >> > From: Lucien Courcol <[email protected]> >> > To: David Lang <[email protected]> >> > Cc: Lucien Courcol via rsyslog <[email protected]> >> > Subject: Re: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops >> working >> > ~10min after starting >> > >> > Hello, >> > >> > So I loaded the impstats.so module and restarted Rsyslog. >> > >> > Rsyslog stopped forwarding logs in less than 5 min after this first >> restart >> > so I didn't get any rsyslogd-pstats logs (I use the default option to >> send >> > stats to the syslog stream with interval set to 5min). >> > >> > I restarted Rsyslog and this time it kept working a bit longer. After 1 >> hr >> > Rsyslog stopped forwarding logs. >> > >> > Filtering on "rsyslogd-pstats" and "main Q" in Splunk gave me this: >> > >> > 2019-09-12T12:15:59.622665+02:00 vcenter rsyslogd-pstats - - - main Q: >> > origin=core.queue size=35 enqueued=6515 full=0 discarded.full=0 >> discarded.nf=0 >> > maxqsize=145 *(last log)* >> > 2019-09-12T12:10:59.521762+02:00 vcenter rsyslogd-pstats - - - main Q: >> > origin=core.queue size=35 enqueued=5599 full=0 discarded.full=0 >> discarded.nf=0 >> > maxqsize=145 >> > 2019-09-12T12:05:59.421025+02:00 vcenter rsyslogd-pstats - - - main Q: >> > origin=core.queue size=35 enqueued=5062 full=0 discarded.full=0 >> discarded.nf=0 >> > maxqsize=145 >> > 2019-09-12T12:00:59.329604+02:00 vcenter rsyslogd-pstats - - - main Q: >> > origin=core.queue size=35 enqueued=4512 full=0 discarded.full=0 >> discarded.nf=0 >> > maxqsize=145 >> > >> > Another filter in Splunk ( "rsyslogd-pstats" submitted NOT >> "origin=imptcp" >> > NOT "origin=imtcp" ) gave me this: >> > >> > 2019-09-12T12:15:59.622648+02:00 vcenter rsyslogd-pstats - - - >> > imudp(*:514): origin=imudp submitted=0 disallowed=0 *(last log) * >> > 2019-09-12T12:15:59.622642+02:00 vcenter rsyslogd-pstats - - - >> > imudp(*:514): origin=imudp submitted=91 disallowed=0 >> > 2019-09-12T12:15:59.622223+02:00 vcenter rsyslogd-pstats - - - imuxsock: >> > origin=imuxsock submitted=6009 ratelimit.discarded=0 >> > ratelimit.numratelimiters=0 >> > 2019-09-12T12:10:59.521748+02:00 vcenter rsyslogd-pstats - - - >> > imudp(*:514): origin=imudp submitted=0 disallowed=0 >> > 2019-09-12T12:10:59.521742+02:00 vcenter rsyslogd-pstats - - - >> > imudp(*:514): origin=imudp submitted=83 disallowed=0 >> > 2019-09-12T12:10:59.521431+02:00 vcenter rsyslogd-pstats - - - imuxsock: >> > origin=imuxsock submitted=5139 ratelimit.discarded=0 >> > ratelimit.numratelimiters=0 >> > 2019-09-12T12:05:59.421011+02:00 vcenter rsyslogd-pstats - - - >> > imudp(*:514): origin=imudp submitted=0 disallowed=0 >> > 2019-09-12T12:05:59.421006+02:00 vcenter rsyslogd-pstats - - - >> > imudp(*:514): origin=imudp submitted=66 disallowed=0 >> > 2019-09-12T12:05:59.420771+02:00 vcenter rsyslogd-pstats - - - imuxsock: >> > origin=imuxsock submitted=4657 ratelimit.discarded=0 >> > ratelimit.numratelimiters=0 >> > >> > * I have the same issue with the another vCenter 6.7U3. * >> > Rsyslog stops forwarding logs after a while. >> > >> > Both have the same Rsyslog version: >> > >> > rsyslogd 8.37.0, compiled with: >> > PLATFORM: x86_64-unknown-linux-gnu >> > PLATFORM (lsb_release -d): >> > FEATURE_REGEXP: Yes >> > GSSAPI Kerberos 5 support: No >> > FEATURE_DEBUG (debug build, slow code): No >> > 32bit Atomic operations supported: Yes >> > 64bit Atomic operations supported: Yes >> > memory allocator: system default >> > Runtime Instrumentation (slow code): No >> > uuid support: Yes >> > systemd support: Yes >> > Number of Bits in RainerScript integers: 64 >> > >> > This is my /etc/rsyslog.conf (identical on both vCenter): >> > >> > >> ################################################################################ >> > ############################# VMware Rsyslog Configuration >> > #################### >> > >> ################################################################################ >> > ###### Module declarations ###### >> > $ModLoad impstats.so >> > $ModLoad imuxsock.so >> > $ModLoad imtcp.so # TLS >> > $ModLoad imptcp.so # TCP >> > $ModLoad imudp.so # UDP >> > $ModLoad omrelp.so # RELP >> > ###### Common configuration ###### >> > $EscapeControlCharactersOnReceive off >> > ###### Template declarations ###### >> > $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log" >> > $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% >> > %app-name% %msg%\n" >> > $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log" >> > $template vpxdFmt,"%msg%\n" >> > $template >> rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log" >> > $template rsyslogadminFmt,"%timestamp:::date-rfc3339% >> %syslogseverity-text% >> > %app-name% %msg%\n" >> > $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log" >> > $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% >> > %hostname% %app-name% %msg%\n" >> > $template defaultSystemLoc,"/var/log/vmware/messages" >> > ###### Rule declarations ###### >> > # TCP/UDP/rsyslog input ruleset declaration >> > $RuleSet all >> > # Make gtls driver the default >> > $DefaultNetstreamDriver gtls >> > # Shared certificate authority certificate >> > $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt >> > # Client certificate >> > $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt >> > # Client key >> > $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key >> > # Include the configuration for syslog relay >> > # _must_ be first to relay all messages >> > $IncludeConfig /etc/vmware-syslog/syslog.conf >> > # vmware services >> > :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmdird" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmcad" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "rbd" ?defaultLoc;defaultFmt >> > & stop >> > :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt >> > & stop >> > :programname, isequal, "vmon" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "pod" stop >> > :programname, isequal, "updatemgr" stop >> > # vpxd-svcs logs to its local logs, hence avoiding duplicate logging. >> > :programname, isequal, "vpxd-svcs" stop >> > # vmware-hvc logs to its local logs, hence avoiding duplicate logging. >> > :programname, isequal, "hvc" stop >> > # vpxd logs to its local logs, hence avoiding duplicate logging. >> > :programname, isequal, "vpxd" stop >> > # For local host's syslog and system logs use the following rules >> > # localhost >> > if $fromhost contains $$myhostname then ?defaultSystemLoc >> > & stop >> > #localhost >> > :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc >> > & stop >> > # ESX rules >> > # Define large LinkedList action queue with 2K msgs cap to accomodate >> 100 >> > ESXs >> > $ActionQueueSize 2000 >> > # Do not choke ESXs, rather start dropping messages after queue is 97.5% >> > full >> > $ActionQueueDiscardMark 1950 >> > $ActionQueueDiscardSeverity 0 >> > $ActionQueueTimeoutEnqueue 1 >> > # VC syslog server log collection >> > *.* ?esxLoc;esxFmt >> > ###### Input server declarations ###### >> > # Setup input flow >> > $DefaultRuleset all >> > $InputPTCPServerBindRuleset all >> > $InputPTCPServerRun 514 >> > $InputUDPServerBindRuleset all >> > $UDPServerRun 514 >> > $InputTCPServerBindRuleset all >> > $InputTCPServerStreamDriverMode 1 >> > $InputTCPServerStreamDriverAuthMode anon >> > $InputTCPServerRun 1514 >> > # >> > # cron log entries for GEN003160 >> > # >> > cron.* -/var/log/cron >> > >> > # >> > # auth.log entries for GEN003660 >> > # >> > auth.* -/var/log/auth.log >> > >> > >> > *However*, in a 3rd vCenter, which is running a slightly different >> version >> > (VCSA 6.7 U2a with rsyslogd 8.15.0), I don't have that issue. >> > >> > This is the /etc/rsyslog.conf of the vCenter where Rsyslog works fine: >> > >> > >> ################################################################################ >> > ############################# VMware Rsyslog Configuration >> > #################### >> > >> ################################################################################ >> > ###### Module declarations ###### >> > $ModLoad imuxsock.so >> > $ModLoad imtcp.so # TLS >> > $ModLoad imptcp.so # TCP >> > $ModLoad imudp.so # UDP >> > $ModLoad omrelp.so # RELP >> > ###### Common configuration ###### >> > $EscapeControlCharactersOnReceive off >> > ###### Template declarations ###### >> > $template defaultLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log" >> > $template defaultFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% >> > %app-name% %msg%\n" >> > $template vpxdLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log" >> > $template vpxdFmt,"%msg%\n" >> > $template >> rsyslogadminLoc,"/var/log/vmware/%app-name%/%app-name%-syslog.log" >> > $template rsyslogadminFmt,"%timestamp:::date-rfc3339% >> %syslogseverity-text% >> > %app-name% %msg%\n" >> > $template esxLoc,"/var/log/vmware/esx/%hostname%/%hostname%-syslog.log" >> > $template esxFmt,"%timestamp:::date-rfc3339% %syslogseverity-text% >> > %hostname% %app-name% %msg%\n" >> > $template defaultSystemLoc,"/var/log/vmware/messages" >> > ###### Rule declarations ###### >> > # TCP/UDP/rsyslog input ruleset declaration >> > $RuleSet all >> > # Make gtls driver the default >> > $DefaultNetstreamDriver gtls >> > # Shared certificate authority certificate >> > $DefaultNetstreamDriverCAFile /etc/vmware/vmware-vmafd/ca.crt >> > # Client certificate >> > $DefaultNetstreamDriverCertFile /etc/vmware/vmware-vmafd/machine-ssl.crt >> > # Client key >> > $DefaultNetstreamDriverKeyFile /etc/vmware/vmware-vmafd/machine-ssl.key >> > # Include the configuration for syslog relay >> > # _must_ be first to relay all messages >> > $IncludeConfig /etc/vmware-syslog/syslog.conf >> > # vmware services >> > :programname, isequal, "applmgmt-audit" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmdird" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmafdd" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmcad" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmdnsd" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "rbd" ?defaultLoc;defaultFmt >> > & stop >> > :app-name, startswith, "rsyslog" ?rsyslogadminLoc;rsyslogadminFmt >> > & stop >> > :programname, isequal, "vmon" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "vmcamd" ?defaultLoc;defaultFmt >> > & stop >> > :programname, isequal, "pod" stop >> > :programname, isequal, "updatemgr" stop >> > # vpxd-svcs logs to its local logs, hence avoiding duplicate logging. >> > :programname, isequal, "vpxd-svcs" stop >> > # vmware-hvc logs to its local logs, hence avoiding duplicate logging. >> > :programname, isequal, "hvc" stop >> > # vpxd logs to its local logs, hence avoiding duplicate logging. >> > :programname, isequal, "vpxd" stop >> > # For local host's syslog and system logs use the following rules >> > # localhost >> > if $fromhost contains $$myhostname then ?defaultSystemLoc >> > & stop >> > #localhost >> > :fromhost-ip, isequal, "127.0.0.1" ?defaultSystemLoc >> > & stop >> > # ESX rules >> > # Define large LinkedList action queue with 2K msgs cap to accomodate >> 100 >> > ESXs >> > $ActionQueueSize 2000 >> > # Do not choke ESXs, rather start dropping messages after queue is 97.5% >> > full >> > $ActionQueueDiscardMark 1950 >> > $ActionQueueDiscardSeverity 0 >> > $ActionQueueTimeoutEnqueue 1 >> > # VC syslog server log collection >> > *.* ?esxLoc;esxFmt >> > ###### Input server declarations ###### >> > # Setup input flow >> > $DefaultRuleset all >> > $InputPTCPServerBindRuleset all >> > $InputPTCPServerRun 514 >> > $InputUDPServerBindRuleset all >> > $UDPServerRun 514 >> > $InputTCPServerBindRuleset all >> > $InputTCPServerStreamDriverMode 1 >> > $InputTCPServerStreamDriverAuthMode anon >> > $InputTCPServerRun 1514 >> > >> > The difference is at the end: >> > >> > The conf file *with the issue* has these lines: >> > >> > # >> > # cron log entries for GEN003160 >> > # >> > cron.* -/var/log/cron >> > >> > # >> > # auth.log entries for GEN003660 >> > # >> > auth.* -/var/log/auth.log >> > >> > >> > I don't really know what that means and if it can explain the issue. >> > >> > Lucien >> > >> > >> > On Wed, Sep 11, 2019 at 10:24 PM Lucien Courcol < >> [email protected]> >> > wrote: >> > >> >> Thx David Lang, >> >> >> >> I'll work on that tomorrow. >> >> >> >> I'll use this guide to get impstats logs: >> >> https://www.rsyslog.com/how-to-use-impstats/ >> >> >> >> Lucien >> >> >> >> On Wed, Sep 11, 2019 at 7:56 PM David Lang <[email protected]> wrote: >> >> >> >>> wihtout seeing your rsyslog.conf we can't begin to guess what's >> >>> happening. >> >>> configuring impstats and showing us the output when it's stopped would >> >>> help us >> >>> to see what's happening. >> >>> >> >>> David Lang >> >>> >> >>> On Wed, 11 Sep 2019, Lucien Courcol via rsyslog wrote: >> >>> >> >>>> Date: Wed, 11 Sep 2019 14:45:38 +0200 >> >>>> From: Lucien Courcol via rsyslog <[email protected]> >> >>>> To: [email protected] >> >>>> Cc: Lucien Courcol <[email protected]> >> >>>> Subject: [rsyslog] Rsyslog in vCenter 6.7U3 (Photon OS) stops working >> >>> ~10min >> >>>> after starting >> >>>> >> >>>> Hello, >> >>>> >> >>>> We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago >> >>> and >> >>>> we noticed a gap of logs in our syslog server (kiwi) since then. >> >>>> >> >>>> I did a bit of troubleshooting but Rsyslog (the syslog client >> running on >> >>>> VCSA) is completely new to me. >> >>>> >> >>>> I use this command to restart Rsyslog: >> >>>> >> >>>> systemctl restart rsyslog >> >>>> >> >>>> Right after starting up Rsyslog, logs are being sent to our syslog >> >>> server. >> >>>> >> >>>> ~10min later, no more logs are sent. >> >>>> >> >>>> The vCenter log file in our syslog server stops getting updated. >> >>>> I did a tcpdump in our vCenter and I see that the vCenter stops >> sending >> >>>> logs. >> >>>> Using UDP or TCP doesn't fix the issue. >> >>>> I looked for errors in various log files in the vCenter but can't >> find >> >>>> anything. >> >>>> >> >>>> This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like >> >>> after >> >>>> restarting Rsyslog >> >>>> >> >>>> 2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin >> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info=" >> >>>> http://www.rsyslog.com";] exiting on signal 15. >> >>>> 2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment >> variable >> >>> TZ >> >>>> is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try >> >>>> http://www.rsyslog.com/e/2442 ] >> >>>> 2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired >> UNIX >> >>>> socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0] >> >>>> 2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin >> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info=" >> >>>> http://www.rsyslog.com";] start >> >>>> >> >>>> Rsyslog is still running based on this command >> >>>> >> >>>> systemctl status rsyslog.service >> >>>> >> >>>> ● rsyslog.service - System Logging Service >> >>>> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; >> >>> vendor >> >>>> preset: enabled) >> >>>> Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min >> ago >> >>>> Docs: man:rsyslogd(8) >> >>>> http://www.rsyslog.com/doc/ >> >>>> Main PID: 22235 (rsyslogd) >> >>>> Tasks: 12 >> >>>> Memory: 5.7M >> >>>> CPU: 191ms >> >>>> CGroup: /system.slice/rsyslog.service >> >>>> └─22235 /usr/sbin/rsyslogd -n >> >>>> >> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: >> Main >> >>>> process exited, code=killed, status=9/KILL >> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System >> Logging >> >>>> Service. >> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: >> Unit >> >>>> entered failed state. >> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: >> Failed >> >>>> with result 'signal'. >> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System >> Logging >> >>>> Service... >> >>>> Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System >> Logging >> >>>> Service. >> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment >> >>> variable >> >>>> TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 >> try >> >>>> http://www.rsyslog.com/e/2442 ] >> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: >> Acquired >> >>>> UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. >> >>> [v8.37.0] >> >>>> Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin >> >>>> software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info=" >> >>>> http://www.rsyslog.com";] start >> >>>> (real hostname has been replaced by vcenter.domain.local) >> >>>> >> >>>> I created a ticket at VMware support, but the agent wasn't able to >> find >> >>> any >> >>>> errors as well and she suggested to take a backup of our vCenter and >> >>>> reinstall with a restore to get a fresh install of Photon OS since >> >>> Rsyslog >> >>>> is integrated in Photon OS. I'm not going to do that now, maybe as a >> >>> last >> >>>> troubleshooting step. >> >>>> >> >>>> In the meantime, do you guys have an idea? Wrong Rsyslog config? >> >>>> >> >>>> Thx for your help. >> >>>> _______________________________________________ >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST if >> >>> you DON'T LIKE THAT. >> >> >> >> >> > > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
