Hi Radu, thanks for the response. I've ran rsyslog on one of the machines, as you specified below. After that I redeployed my logging server and changed the IP. There was no change in the behavior or syslog - I still get the same logs before and after the change. The logs are a little verbose, so here's a snippet of what I see regarding my omfwd action (this is from after the change, but it was looking identical before - except for the timestamp):
5545.609123832:7f1331cdf700: actionCommitAll: action 4, state 1, nbr to commit 0 isTransactional 1 5545.609128757:7f1331cdf700: doTransaction: have commitTransaction IF, using that, pWrkrInfo 0x196d090 5545.609133137:7f1331cdf700: entering actionCallCommitTransaction(), state: itx, actionNbr 4, nMsgs 1 5545.609137473:7f1331cdf700: logging.server 5545.609142140:7f1331cdf700: logging.server:5545/tcp 5545.609147889:7f1331cdf700: omfwd: add 297 bytes to send buffer (curr offs 0) 5545.609152831:7f1331cdf700: omfwd: endTransaction, offsSndBuf 297, iRet -2121 5545.609183045:7f1331cdf700: omfwd: TCP sent 297 bytes, requested 297 5545.609188736:7f1331cdf700: Action 4 transitioned to state: rdy 5545.609193196:7f1331cdf700: omfwd: beginTransaction 5545.609197183:7f1331cdf700: logging.server 5545.609201276:7f1331cdf700: Action 4 transitioned to state: itx 5545.609205543:7f1331cdf700: Action 4 transitioned to state: rdy 5545.609209770:7f1331cdf700: actionCommit, in retry loop, iRet 0 Another thing I figured out - which makes me more troubled about the apparent change of behavior between rsyslog 5.x and 8.x - is that when I deploy a new logging server and assign the elastic IP to it, the IP that rsyslog sees does change: all machines are running on EC2 and so address each other using EC2's "private IPs" (a class A non-routable network), while the elastic IP is a public IP mapped to the actual internal IP of the logging server. So the behavior can be explained by rsyslog not doing DNS lookups for its remote targets: once it starts and resolves the logging.server name to the intenal IP, it keeps sending there, even after the elastic IP has moved. While the old machine is still running (because moving the elastic IP does not change its private IP, nor terminates it) the logs keep showing up on the old machine. Once the old machine is turned off, I can see rsyslog reopening the omfwd connection and everything starts working again. A solution for my setup might be a setting in rsyslog to either always re-resolve the DNS record before submitting a new message, or at least occasionally refresh the cached DNS result (every 60 seconds or so). > From: Radu Gheorghe <[email protected]> > > Hi Oded, > > I've never seen this issue, but maybe you can see something in the debug > log? > > To get the debug log, I usually start rsyslog with something like: > > # rsyslogd -dn > /var/log/rsyslog.log > > Best regards, > Radu > On Mon, Aug 11, 2014 at 4:08 PM, Oded Arbel <[email protected]> wrote: > Hi list, > > I've been using the rsyslog version 5.8 delivered with Ubuntu 12.04 > (which my servers are currently running with) to forward message to a > central logstash server over TCP, and that worked fine - the > forwarding part, I had problems with multiline messages and other > formatting issues, so I upgraded to rsyslog 8 and started to use the > omfwd module with a complex template - and now those problems are > solved, but I have a new one. > > The central logging server is on EC2 with an elastic IP and I update > its configuration from time to time by basically building a new server > and moving the elastic IP. With the old 5.8 installation, this used to > work fine - as soon as the elastic IP moved, all the rsyslog daemons > would lose the connection to the server and reconnect. With version > 8's omfwd, this doesn't happen - rsyslog doesn't notice the connection > dropping and doesn't reconnect. I have to log in to each servers and > restart the service to get it to deliver messages again. > > There are messages sent to the rsyslogds during that time, and these > are logged to the local files correctly, but not delivered to the > central server (neither the old nor the new one). Also, nothing about > this is logged to the local syslog file. > > Please advise? > -- Oded _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
