On Thu, 26 Mar 2015, Peter Viskup wrote:
We just discovered the connection flow issues were caused by central syslog
(no rsyslog running there, but closed-source enterprise solution) which
just fallback to non-TLS when the certificate expires. Looks like the
rsyslog was just waiting for TLS SERVER HELLO message in endless loop.
Question - does newer versions of rsyslog recognize such TLS state properly?
The issue with blocked rsyslog not logging to local messages (server1)
still remains relevant and not solved. Hints are still highly appreciated.
The problem is that when rsyslog can't send the logs remotely, they back up in
the queue. If you don't have an action queue, this stops all processing. If you
have an action queue, other processing continues until the action queue fills
up, at which time the main queue blocks.
you need to either have a large disk queue for the action (but when it fills up
you have the same problem), or you need to configure watermark and discard
settings so that you start throwing logs away when too many are queued up so
that you don't end up blocking.
David Lang
On Wed, Mar 25, 2015 at 4:31 PM, Peter Viskup <skupko...@gmail.com> wrote:
Additional information:
on server1 - rsyslog restarts didn't improve the situation and all the
local logging wasn't working = no changes in local log files. Then I
restarted the rsyslog with disabled forwards in config and the logging just
started working. Then after enablement of forwards the logging is
functioning properly except no communication flow is seen via the TLS
secured connection. Weird.
On Wed, Mar 25, 2015 at 2:44 PM, Peter Viskup <skupko...@gmail.com> wrote:
This seems that we have two servers with similar issues, not sure how
they relate to each other.
The first one with blocked rsyslogd causing issues with writes to
/dev/log (login, sudo, su hang on writes). Complete configuration
server1.txt.
The second one with non-functional TLS secured TCP forward. Complete
configuration in server2.txt.
Debug logs are attached too. Will appreciate any help to discover the
root cause of this issues.
On Wed, Mar 25, 2015 at 10:51 AM, Rainer Gerhards <
rgerha...@hq.adiscon.com> wrote:
I hope I don't have it overlooked, but I haven't seen you config. Can
you please post it.
"itx" means "in transaction", that means the action is within the
middle of a batch. That's quite normal. "rdy" means it's ready to
work, which most probably means it is currently not processing data.
Debug logs unfortunately only help me if they are complete. But I
config shold be enough for starters.
Side-note: heading to a conference this evening, so I'll probably be
unresponsive the next two days.
Rainer
2015-03-25 10:01 GMT+01:00 Peter Viskup <skupko...@gmail.com>:
Thank you, David, for your response.
Today the issue re-occured on other server. In this case the
connection is
established, but no communication is seen at all ( target TLS
encrypted TCP
10.A.B.C:6514 ).
We have two TCP forwards configured for the server - the problematic
FWD is
TLS encrypted, but I was not able to discover any issue with TLS. From
debug log it's obvious the FWDs are in different states, but not sure
what
it does mean (itx vs. rdy).
Would appreciate if somebody could have a look on this and explain what
could be wrong there.
Thank you.
On Thu, Mar 19, 2015 at 4:53 PM, David Lang <da...@lang.hm> wrote:
The "problem" is much more fundamental than you are thinking. It's not
something that rsyslog or any other logging daemon can fix.
The syslog spec says that a write to /dev/log must not complete until
the
log is processed by the syslog daemon. Traditionally this meant that
the
write blocked until the log was written to disk and an fsync
performed.
Rsyslog relaxes this to mean that it only blocks until rsyslog is
able to
read the message and add it to it's internal queue (this is a big
part of
what allows rsyslog to be ~10,000x faster than traditional syslog,
while
doing more with the log).
However, if Rsyslog is unable to output the log message (due to
/var/log
being full, network connection problems, etc), the messages stay in
the
queue and eventually the queue fills up. If the queue is full, rsyslog
cannot read the message from /dev/log and anything trying to write to
it
will hang.
By default, these queues are held in RAM. They can be changed in
size, you
can make a separate queue for the network connection, and you can
configure
rsyslog to use disk for some of it's queue space. But no matter what
you
do, a long enough outage of a network connection will fill up the
queues
and processing will stop.
At this point you have to ask yourself, which would you rather have
happen, loose some logs or stop processing.
If you are willing to loose some logs when you have big enough
problems
with logging, then there are several options available to you
I recommend that you use UDP for the first hop from a system to a log
relay on the same network. This is extremely reliable (yes, UDP can be
lost, but it's lost when there is contention at a bottleneck, if your
only
bottleneck is the backplane of the switch, you are extremely unlikely
to
loose any log messages due to this)
You can also configure high watermark levels in rsyslog that will
change
it's behavior when there are too many messages in the queue and start
throwing away low priority messages. It's important to note that this
is
only delaying the problem, eventually the queues will still fill up
with
the high priority messages.
If you do decide that you would rather stop processing rather than
loosing
messages, then you need to be aware that simple TCP forwarding is not
sufficient to avoid loosing messages when there are network outages.
You
will need to use RELP (reliable event logging protocol) to avoid
message
loss when there are network hiccups. And after that there is the
problem of
the messages held in the memory queues that will be lost if a system
reboots
none of this is a problem with rsyslog, it's fundamental to the
problem of
logging. No matter what tool you use for your logs, what should
happen when
you can't deliver the log? If you configure the system to not throw
the log
away, you are implicitly telling the system that you would rather stop
until it can proces the log message.
David Lang
On Thu, 19 Mar 2015, Peter Viskup wrote:
Hi we discovered strange issue on Debian servers with rsyslog
forwarding
messages to remote server via TCP.
Once having TCP forward connection in freeze on CLOSE_WAIT on rsyslog
'client' side the rsyslog hung and stops reading /dev/log and other
sources. This is consequently causing issues with all commands
writing to
/dev/log like su, login or sudo and all the logging not working.
Could anybody check where the issue could be?
System: Debian Wheezy
Rsyslog version: 5.8.11-3 (from wheezy)
This was discovered more times before. Attached is some debug
information
I
was able to reach prior rsyslog restart.
Would appreciate any meaningful comments which could lead to
solution or
workaround.
At the moment it seems that the rsyslog client doesn't handle TCP
sockets
properly which leads to DoS.
Could be related to the known issue described by Rainer in 2010 [1]?
If
yes, then what's the status with current version and since when it
was
fixed?
[1]
http://lists.adiscon.net/pipermail/rsyslog/2010-January/026395.html
Best regards,
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
