Hi David,
I disabled log rotation during my investigation, and this was not the cause.
A cause won't be SELinux because we use Solaris 11.
Impstats module is not available out of the box for Solaris. We don't have this
or the ability to compile in production.
Here is the configuration, in case this helps. The json-template0 template is
defined but not used. json-template2 is used later on.
Also the Local template is not used and the rsyslog server logs its own
messages via DynaFile just like the clients.
The legacy syntax is used because I cannot the new rsyslog syntax.
$ModLoad imsolaris # for Solaris kernel logging
$ModLoad imtcp
$ModLoad imudp
$MainMsgQueueSize 1000
$InputTCPMaxSessions 2000
$InputTCPMaxListeners 20
$WorkDirectory /var/spool/rsyslog/work
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
$template DynaFile,"/var/spool/rsyslog/%HOSTNAME%.log"
template(name="json-template0"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
$template json-template2,
"{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
$FileOwner root
$FileGroup uxadmin
$FileCreateMode 0640
$DirCreateMode 0750
$Umask 0022
$RuleSet Local
*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages
authpriv.* /var/adm/authpriv.log
*.emerg :omusrmsg:*
*.alert :omusrmsg:root
& stop
$RuleSet Remote
*.info ?DynaFile
*.info @@(o)b111l:10514;json-template2
*.info @@(o)b112l:10514;json-template2
& stop
$DefaultRuleset Remote
$InputTCPServerBindRuleset Remote
$InputTCPServerRun 514
$InputUDPServerBindRuleset Remote
$UDPServerRun 514
Best wishes,
Sophie
Team mailbox : [email protected]
or direct [email protected]
> -----Original Message-----
> From: David Lang [mailto:[email protected]]
> Sent: Monday, April 23, 2018 10:27 PM
> To: sophie.loewenthal--- via rsyslog
> Cc: LOEWENTHAL Sophie
> Subject: Re: [rsyslog] server received messages but rsyslog did not write them
> into a file
>
> On Mon, 23 Apr 2018, sophie.loewenthal--- via rsyslog wrote:
>
> > After 31st March our rsyslog v8.4.2 Solaris 11 servers stopped processing
> > lots
> of messages. I think we had a network change but do not yet know what.
> >
> > Tcpdump showed the test message arrived,
> >
> > # tcpdump -s 0 -A -vvv port 514 |grep sdfasdfsa
> > dropped privs to nobody
> > tcpdump: listening on ipmp0, link-type IPNET (Solaris ipnet), capture size
> 262144 bytes
> > .KB.....78 <13>Apr 23 15:01:54 be-s0784-z1a emerg.info: [ID 702911
> user.notice] sdfasdfsa
> > 46 packets captured
> >
> > But the log file for this server remains empty. The rsyslog client sent the
> message over TCP with,
> >
> > *.info @@(o)sysl1:514
> > *.info @@(o)sysl2:514
> >
> > Could this be a routing issue? Or something else.
>
> It could be a lot of things, the most common is that the output logfile was
> moved (or deleted), but rsyslog wasn't sent the HUP signal and so is still
> writing to the old file.
>
> you can use lsof to look for all files that rsyslog has filehandles for and
> see
> what it says.
>
> The next most common cause is that some other output is blocked, and so
> messages
> are just queuing up instead of being written, diagnosing this requires
> enabling
> impstats in rsyslog, and since that requires a restart, the restart will
> probably 'solve' the problem in the short term. If we could see the entire
> config file (and anything included into it), we could make an educated guess
> as
> to if that is the problem or not.
>
> It could also be SELinux problems, but that doesn't usually start happening
> after the system has been running sucessfully
>
> and there is a chance that it's something at the IP layer (routing or
> iptables),
> but again, those don't usually show up after a system is running successfully.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the
internet
cannot guarantee the integrity of this message which may not be reliable, BNP
PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed
or falsified.
Do not print this message unless it is necessary, consider the environment.
----------------------------------------------------------------------------------------------------------------------------------
Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans
l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
