Upgrade to a current version, there have been massive changes in this area
between v5 and v7
David Lang
On Thu, 13 Mar 2014, Tanky Woo wrote:
Date: Thu, 13 Mar 2014 19:50:32 +0800
From: Tanky Woo <wtq1...@gmail.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: rsyslog@lists.adiscon.com
Subject: [rsyslog] rsyslog v5 blocked by dns timeout
I have do another test today:
The network environment is:
rsyslog config a udp translate with domain:
*.* @xxx.com:514
a host with private network 10.0.0.0/16, default route is 10.0.0.99, will go by
this net.
and my network with above it is 10.0.0.0/8, and name server is 10.10.0.100, if
my network in exist, my intra domain xxx.com is resolved by this name server
through 10.0.0.0/8 net.
but now my network have problem and not exist, so the name resolve will through
10.0.0.0/16, and this net will drop my query, cause timeout.
in command host and nslookup, when timeout, the return code is 1, if domain not
exists or ok, return code is 0.
then, if my network 10.0.0.0/8 is ok previous, and rsyslog is ok; now I delete
my network route, and rsyslog is still ok, but when I restart rsyslog, that
will no longer writer log into file.
I change name server to 8.8.8.8, this query package will not be dropped, and
query my intra domain is not exists, return 0, and rsyslog is ok.
So, I think there are some problems to treat domain resolved with timeout in
rsyslog
On Mon, 10 Mar 2014, Tanky Woo wrote:
>
>> On Mon, 10 Mar 2014, Tanky Woo wrote:
>>
>> >> On Sun, 9 Mar 2014, Tanky Woo wrote:
>> >>
>> >> >> On Sun, 9 Mar 2014, Tanky Woo wrote:
>> >> >>
>> >> >> > rsyslog version is 5.8
>> >> >> >
>> >> >> > I found rsyslogd will be blocked, if the remote server domain name
can’t resolve because of timeout.
>> >> >> >
>> >> >> > I configured the rsyslogd to send log to remote by udp:
>> >> >> >
>> >> >> > *.* @xxx.com:514
>> >> >> >
>> >> >> > now the name server has problem and the dns request will
timeout(only timeout will cause this problem). the cause the rsyslogd can’t work right.
>> >> >> >
>> >> >> > I use logger to send log, and no log will write in /var/log/syslog。
>> >> >> >
>> >> >> > I just add the domain in /etc/hosts to solve this problem. But is
there any other way, or is it a bug in v5?
>> >> >>
>> >> >> The message where the timeout happens will eventually get into the log, but the
>> >> >> problem is that while it is stalled waiting for the timeout, other UDP messages
>> >> >> will end up getting dropped.
>> >> >>
>> >> >> you can disable name resolution entirely by starting rsyslog with the -x flag
>> >> >> (it won't even use /etc/hosts at that point)
>> >> >>
>> >> >> This is a problem with basically every syslog daemon from the beginning. Rsyslog
>> >> >> 7+ has a DNS cache, but I'm not sure that it will cach failures, so you may
>> >> >> still have a problem (I think it also changes the point at which the DNS
>> >> >> resolution happens, so it may receive message without a problem, but then get
>> >> >> stuck writing them out)
>> >> >>
>> >> >> David Lang
>> >> > I have already setted the `-x` options:
>> >> > RSYSLOGD_OPTIONS="-c5 -Q -x”
>> >> > and it will still blocked.
>> >> > and the manual say:
>> >> > -x Disable DNS for remote messages.
>> >> > if I disable it, it will not do name resolution, how can the rsyslog to
use xxx.com in configuration?
>> >>
>> >> It depends how you are using xxx.com in your configuration. If you disable name
>> >> resolution, you don't get %fromhost%, but you still have %fromhost-ip% and if
>> >> the sender sets %hostname% in the message you can still use that.
>> >>
>> >> > then:
>> >> >> The message where the timeout happens will eventually get into the
log, but the problem is that while it is stalled waiting for the timeout, other UDP messages will
end up getting dropped.
>> >> >
>> >> > If it is stalled waiting for the timeout, and the udp messages will
dropped, why rsyslog won’t write message to file, I just move that statements to the end of
the configuration.
>> >>
>> >> the message should end up getting to the file eventually, it's other messages
>> >> that arrive during this time that will end up being dropped. This happens as
>> >> rsyslog is receiving the message to put it into the queue to be processed, it
>> >> doesn't matter where in the configuration the output is.
>> >>
>> >> David Lang
>> >
>> > So, if the dns request timeout, the message will in the queue and block the
>> > queue? I didn’t configured Main Queue, the default mode is direct mode. and
>> > this will also block queue?
>>
>> back in 5.8 (remember that we are on 7.6 and 8.x now), the UDP listener would
>> receive the message, do a DNS lookup, then when it succeeded or timed out it
>> owuld put the message in the main queue and it would be processed with no
>> further delay
>>
>> The problem is that while the UDP listener was stalled waiting for the DNS
>> lookup to timeout, if too many additional packets arrived on the system, they
>> would overflow the OS buffer and be dropped before rsyslog got around to asking
>> for them
>>
>> > I have tried to configured main queue as DA type , but there is no use.
>>
>> Right, this won't make any difference at all. The problem is in the receiver,
>> before it gets put into any queue at all.
>>
>> David Lang
> Sorry， I said wrong, the default type of main queue is fixed array，not direct。
>
> I think maybe there is misunderstand in the question, my local system use
rsyslog to send log to the remote syslog machine.
>
> so the message put into main queue, but cannot get out to output to send. not
the udp listener being stalled waiting for the dns lookup. and I think it is not
the problem of the receiver, but the sender.
>
once it is in the main queue, you should have no further problems sending it. At
that point the name lookup has completed (success or failure).
The only problem you could have is if the machine you are relaying it to can't
find your relay box in it's name lookup.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
