Hi Radu, thanks for the response.
I've ran rsyslog on one of the machines, as you specified below. After that
I redeployed my logging server and changed the IP. There was no change in
the behavior or syslog - I still get the same logs before and after the
change. The logs are a little verbose, so here's a snippet of what I see
regarding my omfwd action (this is from after the change, but it was
looking identical before - except for the timestamp):
5545.609123832:7f1331cdf700: actionCommitAll: action 4, state 1, nbr to
commit 0 isTransactional 1
5545.609128757:7f1331cdf700: doTransaction: have commitTransaction IF,
using that, pWrkrInfo 0x196d090
5545.609133137:7f1331cdf700: entering actionCallCommitTransaction(), state:
itx, actionNbr 4, nMsgs 1
5545.609137473:7f1331cdf700: logging.server
5545.609142140:7f1331cdf700: logging.server:5545/tcp
5545.609147889:7f1331cdf700: omfwd: add 297 bytes to send buffer (curr offs
0)
5545.609152831:7f1331cdf700: omfwd: endTransaction, offsSndBuf 297, iRet
-2121
5545.609183045:7f1331cdf700: omfwd: TCP sent 297 bytes, requested 297
5545.609188736:7f1331cdf700: Action 4 transitioned to state: rdy
5545.609193196:7f1331cdf700: omfwd: beginTransaction
5545.609197183:7f1331cdf700: logging.server
5545.609201276:7f1331cdf700: Action 4 transitioned to state: itx
5545.609205543:7f1331cdf700: Action 4 transitioned to state: rdy
5545.609209770:7f1331cdf700: actionCommit, in retry loop, iRet 0
Another thing I figured out - which makes me more troubled about the
apparent change of behavior between rsyslog 5.x and 8.x - is that when I
deploy a new logging server and assign the elastic IP to it, the IP that
rsyslog sees does change: all machines are running on EC2 and so address
each other using EC2's "private IPs" (a class A non-routable network),
while the elastic IP is a public IP mapped to the actual internal IP of the
logging server.
So the behavior can be explained by rsyslog not doing DNS lookups for its
remote targets: once it starts and resolves the logging.server name to the
intenal IP, it keeps sending there, even after the elastic IP has moved.
While the old machine is still running (because moving the elastic IP does
not change its private IP, nor terminates it) the logs keep showing up on
the old machine. Once the old machine is turned off, I can see rsyslog
reopening the omfwd connection and everything starts working again.
A solution for my setup might be a setting in rsyslog to either always
re-resolve the DNS record before submitting a new message, or at least
occasionally refresh the cached DNS result (every 60 seconds or so).
From: Radu Gheorghe <[email protected]>
Hi Oded,
I've never seen this issue, but maybe you can see something in the debug
log?
To get the debug log, I usually start rsyslog with something like:
# rsyslogd -dn > /var/log/rsyslog.log
Best regards,
Radu
On Mon, Aug 11, 2014 at 4:08 PM, Oded Arbel <[email protected]> wrote:
Hi list,
I've been using the rsyslog version 5.8 delivered with Ubuntu 12.04
(which my servers are currently running with) to forward message to a
central logstash server over TCP, and that worked fine - the
forwarding part, I had problems with multiline messages and other
formatting issues, so I upgraded to rsyslog 8 and started to use the
omfwd module with a complex template - and now those problems are
solved, but I have a new one.
The central logging server is on EC2 with an elastic IP and I update
its configuration from time to time by basically building a new server
and moving the elastic IP. With the old 5.8 installation, this used to
work fine - as soon as the elastic IP moved, all the rsyslog daemons
would lose the connection to the server and reconnect. With version
8's omfwd, this doesn't happen - rsyslog doesn't notice the connection
dropping and doesn't reconnect. I have to log in to each servers and
restart the service to get it to deliver messages again.
There are messages sent to the rsyslogds during that time, and these
are logged to the local files correctly, but not delivered to the
central server (neither the old nor the new one). Also, nothing about
this is logged to the local syslog file.
Please advise?
