Re: strange problem with query being dropped/ignored by the BIND process

[Marc Richter]

Hi again,

I have checked this again today.

Send & receive buffers are both 1MB, the Server has 8 CPUs and during
startup BIND is reporting this:

found 8 CPUs, using 8 worker threads
using 7 UDP listeners per interface
using up to 32768 sockets

We only have about 1.500 queries per second on this server. CPU(30%) and
memory(50%) usage also is not an issue here.

Now Oracle support is saying that the buffer sizes are fine and we need to
"speed up the application" to read the data faster from the receive buffer
and this prevent package drops.

Do you think that is a reasonable statement in this environment ?
What would be the best way to "speed up the application" ? Just increase
the worker threads ?

Regards
Marc


On 06/28/17 15:31, Marc Richter wrote:
> Hi Ben,
> 
> thanks for the answer.
> 
> Yeah, I think you are right. I see a lot of udpInOverflows on the system,
> which suggest that the receive buffer is too small indeed.
> 
> Is there any kind of recommendation or best-practice advice what the
> buffers should ideally be set to on Solaris ?
> I did search the ISC Knowledge Base, but didn't find any useful advice.
> 
> Regards
> arc
> 
> On 06/28/17 14:37, Ben Croswell wrote:
>> Have you checked deeper at the OS level? I have seen on Linux DNS servers
>> silent drops of queries on very busy servers that were exhausting UDP
>> receive buffers.
>>
>> On Jun 28, 2017 10:26 AM, "Marc Richter" > > wrote:
>>
>> Hi,
>>
>> we have a setup here consisting of a recursive DNS server and two
>> monitoring servers. The monitoring servers sent a test query to the DNS
>> server once every two minutes to check if it is answering properly.
>>
>> We now have the problems that these test queries are timing out from time
>> to time, (correctly) resulting in alarms in our monitoring system.
>>
>> I have checked this now and noticed that each time we see that alarm, the
>> query sent by the monitoring server is not being answered at all.
>> To debug that I ran tcpdump on both the monitoring server and the 
>> recursive
>> DNS server. I see the query being sent out on the monitoring server and I
>> also see the query being received on the DNS server, however there is no
>> response sent to this query at all.
>> Looking at the query log, which I enabled temporarily, the query is also
>> not logged there so it looks like BIND is ignoring that query somewhere,
>> although it is properly received by the IP stack of the server.
>>
>> Do you have any suggestions how to debug this further, to hopefully find
>> out where these queries are stuck/dropped/ignored, as I have run out of
>> ideas ?
>>
>> The environment is:
>> BIND 9.9.9-P5 (Extended Support Version) 
>> running on SunOS sun4v 5.11 11.3
>>
>>
>> Thanks !
>> Marc
>> ___
>> Please visit 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.isc.org_mailman_listinfo_bind-2Dusers&d=DwICAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=wDgZv-d1RrBMzWr_7pSF_09ZAXIr59EgoXQU4ctOHMk&m=b8p_t6atDvFHu2tWe4Jgw_EvLufZakMUJL0w06aA3V0&s=bXYnQq1IzLGZG6xbey81qsaTVpqiLVlwxazV8CXVP_A&e=
>>  
>> 
>> 
>> to unsubscribe from this list
>>
>> bind-users mailing list
>> bind-users@lists.isc.org 
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.isc.org_mailman_listinfo_bind-2Dusers&d=DwICAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=wDgZv-d1RrBMzWr_7pSF_09ZAXIr59EgoXQU4ctOHMk&m=b8p_t6atDvFHu2tWe4Jgw_EvLufZakMUJL0w06aA3V0&s=bXYnQq1IzLGZG6xbey81qsaTVpqiLVlwxazV8CXVP_A&e=
>>  
>> 
>> 
>>
>>
> 

-- 
Marc Richter
Engr III Cslt-Ntwk Eng&Ops

Sebrathweg 20
44149 Dortmund
Germany

O +49 231 972 1293
F +49 231 972 2587
E marc.rich...@de.verizon.com
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: strange problem with query being dropped/ignored by the BIND process

[Dennis Clarke]

On 06/29/2017 12:52 PM, Marc Richter wrote:

Hi again,

I have checked this again today.

Send & receive buffers are both 1MB, the Server has 8 CPUs and during
startup BIND is reporting this:

found 8 CPUs, using 8 worker threads
using 7 UDP listeners per interface
using up to 32768 sockets

We only have about 1.500 queries per second on this server. CPU(30%) and
memory(50%) usage also is not an issue here.


Do you have any adjustments in /etc/system ?

I will assume you don't have ip_forwarding messed with and let's just 
look at your network stack config. You don't need to publish your 
results to the maillist but have a look at :


# ndd -get /dev/ip \? | grep "read"
# ndd -get /dev/tcp \? | grep "read"

Here you have the full range of stack kernel tunables. At the very least
the ones you can read data from.

You probably already did this but create a quick script :

#!/bin/sh
/usr/bin/printf "\n"

/usr/bin/printf "tcp_wscale_always = "
ndd -get /dev/tcp tcp_wscale_always

/usr/bin/printf "tcp_tstamp_if_wscale = "
ndd -get /dev/tcp tcp_tstamp_if_wscale

/usr/bin/printf "tcp_max_buf = "
ndd -get /dev/tcp tcp_max_buf

/usr/bin/printf "tcp_cwnd_max = "
ndd -get /dev/tcp tcp_cwnd_max

/usr/bin/printf "tcp_xmit_hiwat = "
ndd -get /dev/tcp tcp_xmit_hiwat

/usr/bin/printf "tcp_recv_hiwat = "
ndd -get /dev/tcp tcp_recv_hiwat


Run that.


What I see here on three diff Sol10 servers for various purposes is :

M5 # /tmp/foo.sh

tcp_wscale_always = 1
tcp_tstamp_if_wscale = 1
tcp_max_buf = 1048576
tcp_cwnd_max = 1048576
tcp_xmit_hiwat = 49152
tcp_recv_hiwat = 49152


st0 # /tmp/foo.sh

tcp_wscale_always = 1
tcp_tstamp_if_wscale = 1
tcp_max_buf = 1048576
tcp_cwnd_max = 1048576
tcp_xmit_hiwat = 49152
tcp_recv_hiwat = 49152


st1 #

tcp_wscale_always = 1
tcp_tstamp_if_wscale = 1
tcp_max_buf = 16777216
tcp_cwnd_max = 8388608
tcp_xmit_hiwat = 65535
tcp_recv_hiwat = 65535


The first two are defaults whereas the last unit needs to sling around
terabytes daily.  I am curious what your system thinks it is doing
with its tcp/ip stack.

Since you are on contract ( me too .. arn't we all these days ) then I
have to assume you have reasonable kernel updates and tcp patches in
this Solaris server ?

Dennis




___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: strange problem with query being dropped/ignored by the BIND process

[Marc Richter]

Hi Dennis,

> Do you have any adjustments in /etc/system ?

No. And as mentioned before this is a Solaris 11 system, so /etc/system is
(mostly) irrelevant, as the IP settings are all done with ipadm now.

> 
> # ndd -get /dev/ip \? | grep "read"
> # ndd -get /dev/tcp \? | grep "read"
> 

That, as well as the script and examples you provided, won't help me a lot,
as I am looking at UDP receive buffer overflows, not TCP.

I have set udp_max_buf to 4MB now and udp_send_buf & udp_recv_buf to 2MB
each, then restarted BIND.
It seems to be working better now as I don't see that much receive buffer
overflows anymore.

However, the initial question still stands. How can I reconfigure BIND to
pick up the data faster from the receive buffer ?

> Since you are on contract ( me too .. arn't we all these days ) then I
> have to assume you have reasonable kernel updates and tcp patches in
> this Solaris server ?

Yes, of course.

Regards
Marc
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

RPZ zone name label length limit

[Jim Yang]

Hi,

What is the DNS name label length limit? As per RFC 1035, it is 63 characters.  
I tested a few DNS names that contains a label that is longer than 63 
characters, and found that these records were successfully loaded in RPZ zone. 
I wonder if this is a BIND RPZ feature or bug (it allows DNS name label that is 
longer than 63 characters)?

When I dig these DNS records using 8.8.8.8, which reports them as ‘NXDOMAIN’.

Thanks,
Jim
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: RPZ zone name label length limit

[Tony Finch]

Jim Yang  wrote:
>
> What is the DNS name label length limit? As per RFC 1035, it is 63
> characters.  I tested a few DNS names that contains a label that is
> longer than 63 characters, and found that these records were
> successfully loaded in RPZ zone.

On the wire the length limit is 63. In presentation format some characters
have to be \escaped which can make the name up to four times longer.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/  -  I xn--zr8h punycode
Plymouth: Northwest 5 to 7, occasionally gale 8 later. Moderate or rough,
occasionally very rough later in west. Occasional rain. Good, occasionally
moderate.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: RPZ zone name label length limit

[Mukund Sivaraman]

Hi Jim

On Thu, Jun 29, 2017 at 01:57:16PM +, Jim Yang wrote:
> Hi,
> 
> What is the DNS name label length limit? As per RFC 1035, it is 63
> characters.  I tested a few DNS names that contains a label that is
> longer than 63 characters, and found that these records were
> successfully loaded in RPZ zone. I wonder if this is a BIND RPZ
> feature or bug (it allows DNS name label that is longer than 63
> characters)?
> 
> When I dig these DNS records using 8.8.8.8, which reports them as
> ‘NXDOMAIN’.

Can you send us a bug report with a sample RPZ zone that contains such a
name?

Mukund
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: RPZ zone name label length limit

[Jim Yang]

Hi Mukund,

Yes, I will send the report with a sample RPZ zone that contains the name to 
bind-b...@isc.org.

Thanks,
Jim

On 6/29/17, 2:40 PM, "Mukund Sivaraman"  wrote:

Hi Jim

On Thu, Jun 29, 2017 at 01:57:16PM +, Jim Yang wrote:
> Hi,
> 
> What is the DNS name label length limit? As per RFC 1035, it is 63
> characters.  I tested a few DNS names that contains a label that is
> longer than 63 characters, and found that these records were
> successfully loaded in RPZ zone. I wonder if this is a BIND RPZ
> feature or bug (it allows DNS name label that is longer than 63
> characters)?
> 
> When I dig these DNS records using 8.8.8.8, which reports them as
> ‘NXDOMAIN’.

Can you send us a bug report with a sample RPZ zone that contains such a
name?

Mukund


___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: strange problem with query being dropped/ignored by the BIND process

[Bob Harold]

On Thu, Jun 29, 2017 at 9:51 AM, Marc Richter 
wrote:

> Hi Dennis,
>
> > Do you have any adjustments in /etc/system ?
>
> No. And as mentioned before this is a Solaris 11 system, so /etc/system is
> (mostly) irrelevant, as the IP settings are all done with ipadm now.
>
> >
> > # ndd -get /dev/ip \? | grep "read"
> > # ndd -get /dev/tcp \? | grep "read"
> >
>
> That, as well as the script and examples you provided, won't help me a lot,
> as I am looking at UDP receive buffer overflows, not TCP.
>
> I have set udp_max_buf to 4MB now and udp_send_buf & udp_recv_buf to 2MB
> each, then restarted BIND.
> It seems to be working better now as I don't see that much receive buffer
> overflows anymore.
>
> However, the initial question still stands. How can I reconfigure BIND to
> pick up the data faster from the receive buffer ?
>
> > Since you are on contract ( me too .. arn't we all these days ) then I
> > have to assume you have reasonable kernel updates and tcp patches in
> > this Solaris server ?
>
> Yes, of course.
>
> Regards
> Marc
>

I tend to distrust  "CPU(30%)" if it is averaged over more than one cpu.
Could you run "top" and hit the number "1" so that it shows each cpu
separately?  With 8 cpu's, "30%" could be one cpu at 100% and others lower,
where the one cpu at 100% is your bottleneck.

Just a guess at something to look at.

-- 
Bob Harold
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: [E] Re: strange problem with query being dropped/ignored by the BIND process

[Marc Richer]

Hi Bob,

> I tend to distrust  "CPU(30%)" if it is averaged over more than one
> cpu.  Could you run "top" and hit the number "1" so that it shows each
> cpu separately?  With 8 cpu's, "30%" could be one cpu at 100% and others
> lower, where the one cpu at 100% is your bottleneck.

I checked that with mpstat earlier already and the load is evenly
distributed amongst all CPUs. None of the CPUs is overloaded.

Regards
Marc

___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

CVE-2017-3142 and CVE-2017-3143 -- TSIG-related BIND vulnerabilities

[Michael McNally]

Today ISC announced two significant BIND vulnerabilities (via our
bind-announce list -- https://lists.isc.org/mailman/listinfo/bind-announce)

They are CVE-2017-3142 and CVE-2017-3143 and both are related to
errors in our TSIG support.  These are unusual CVEs for BIND --
many of the vulnerabilities we disclose are denial-of-service
vectors which affect server availability but can easily be
partly or completely mitigated by running BIND with a watchdog
process.  Atypically, these new vulnerabilities have, respectively,
a confidentiality impact (for CVE-2017-3142, which potentially
permits unauthorized zone transfer) and a data integrity impact
(CVE-2017-3143, which under some circumstances can permit an
attacker to cause the server to accept a forged DDNS update.)

New versions of BIND have been released and are available from
ISC's web site:  http://www.isc.org/downloads

Details on the vulnerabilities are available via the ISC Knowledge Base:
https://kb.isc.org/category/74/0/10/Software-Products/BIND9/Security-Advisories/

Please take these bugs seriously and act promptly to safeguard
your servers if you rely on TSIG authentication for zone transfers
or DDNS.


Michael McNally
ISC Support
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

RHEL, Centos, Fedora rpm 9.11.1-P2

[Carl Byington]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

http://www.five-ten-sg.com/mapper/bind contains links to the source
rpms, and build instructions.


-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEAREKAAYFAllVdXcACgkQL6j7milTFsG/SQCggBDFBEwmgOb92nESct8cg3IS
gOoAn2KXPunBCbmHxvcabF0LqXtcpCUU
=y5IX
-END PGP SIGNATURE-


___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users