Re: Performance Tuning RHEL 5 and Bind
OK I have the source of the problem now I just need an elegant way to fix it and most cost ( Network TCP ) effective way to fix it The Windows Domain is responsible for X.internal.example.com and I am presently forwarding X.internal.example.com to their nameservers DC, resulting in TCP queries. Which is dragging the cache server down when PC's query for records off of [NAME].internal.example.com. I don't mind not caching X.internal.example.com so can I create an NS record or an stub entry that points the PC's else where rather than forwarding them or caching them? Thank You, Brett On Tue, Oct 22, 2013 at 9:39 PM, Alan Clegg a...@clegg.com wrote: On Oct 22, 2013, at 8:29 PM, brett smith brett.s9...@gmail.com wrote: Yes tuning off IPTABLES conn-tracking makes a huge difference. I also followed: https://access.redhat.com/site/solutions/304713 https://access.redhat.com/site/solutions/168483 I still see some SYN_SENT from Windows PC's on tcp port 53 on the DNS cache server. You've cured the symptoms, not the illness. You really, REALLY need to figure out why your clients are doing TCP. You'll see a world of difference when you solve this part of the puzzle. AlanC -- Alan Clegg | +1-919-355-8851 | a...@clegg.com ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
On Oct 28, 2013, at 8:08 PM, brett smith brett.s9...@gmail.com wrote: OK I have the source of the problem now I just need an elegant way to fix it and most cost ( Network TCP ) effective way to fix it The Windows Domain is responsible for X.internal.example.com and I am presently forwarding X.internal.example.com to their nameservers DC, resulting in TCP queries. Which is dragging the cache server down when PC's query for records off of [NAME].internal.example.com. I don't mind not caching X.internal.example.com so can I create an NS record or an stub entry that points the PC's else where rather than forwarding them or caching them? Slave X.internal.example.com AlanC -- Alan Clegg | +1-919-355-8851 | a...@clegg.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
Hi— On Oct 28, 2013, at 9:05 PM, Alan Clegg a...@clegg.com wrote: Slave X.internal.example.com +1; it’s also worth looking into why there is such a high volume of DNS queries. Is it simply a big network with a lot of chatty clients? Or is TTL turned down so low that client side caching is not effective and needs to requery often? Or is something doing a host scan? If it’s your network IDS or security/network admin folks running a portscan, fine; if it’s malware or an intruder scanning the local subnet(s), one might want to notice and take steps to solve the problem rather than a symptom. Regards, — -Chuck ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
Hi, Kevin Darcy k...@chrysler.com writes: Are these queries mostly for names in an Active Directory domain? The default for Active Directory is for *every* Domain Controller to register NS records at the apex of the AD domain. Pretty soon, for any reasonably-sized AD infrastructure, all of those NSes cause *all* queries for *any* name in the domain to trigger a TCP retry (because the Answer + Authority Sections overflow 512 bytes), if EDNS0 is not in effect. I sat down with our AD folks a few years ago and impressed upon them how important it is to be selective about which Domain Controllers are registered at the apex. They appreciated the negative consequences of being awash in TCP retries, and it's been managed for some time now (at least for our *main* AD domain; don't get me started on the business partner that still has 92 NS records at the apex of their AD domain. Sigh) good point. Increasing the EDNS0 UDP size might also be an option (default is 1280 for Windows DNS) - http://technet.microsoft.com/en-us/library/cc783893%28v=ws.10%29.aspx It is possible to tell some less critical DC to not register themself in DNS: http://support.microsoft.com/kb/198767 and http://technet.microsoft.com/en-us/library/cc782946%28v=ws.10%29.aspx -- Carsten ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
On Oct 21, 2013, at 9:47 AM, wbr...@e1b.org wrote: From: Alan Clegg a...@clegg.com Fix your windows clients. You can't fix stupid. I have lots of windows clients and they don't exhibit this feature. There's something wrong on the windows clients and it's not the norm. To be honest, recent windows releases do a pretty fine job with DNS. AlanC -- Alan Clegg | +1-919-355-8851 | a...@clegg.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
-Original Message- From: Alan Clegg a...@clegg.com Date: Tuesday, October 22, 2013 7:44 AM To: bind-users@lists.isc.org bind-users@lists.isc.org Subject: Re: Performance Tuning RHEL 5 and Bind On Oct 21, 2013, at 9:47 AM, wbr...@e1b.org wrote: From: Alan Clegg a...@clegg.com Fix your windows clients. You can't fix stupid. I have lots of windows clients and they don't exhibit this feature. There's something wrong on the windows clients and it's not the norm. To be honest, recent windows releases do a pretty fine job with DNS. Agreed. The problem here is the TCP fall-back vs BIND/OS tuning. I've got a lot of Windows clients (mostly vmware related infra) that don't query via TCP. I would focus on a deeper inspection of the environment including network layer. The OP needs to figure out why the queries are using TCP. Speculating based on the available data, I'm wondering if the new BIND servers were stood up behind a firewall...possibly with broken protocol inspection/fixup type configuration limiting UDP packet size to 512 bytes...and zone data with large NS/whatever RR sets resulting in TCP retries. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
Are these queries mostly for names in an Active Directory domain? The default for Active Directory is for *every* Domain Controller to register NS records at the apex of the AD domain. Pretty soon, for any reasonably-sized AD infrastructure, all of those NSes cause *all* queries for *any* name in the domain to trigger a TCP retry (because the Answer + Authority Sections overflow 512 bytes), if EDNS0 is not in effect. I sat down with our AD folks a few years ago and impressed upon them how important it is to be selective about which Domain Controllers are registered at the apex. They appreciated the negative consequences of being awash in TCP retries, and it's been managed for some time now (at least for our *main* AD domain; don't get me started on the business partner that still has 92 NS records at the apex of their AD domain. Sigh) Sounds like you might need to have the same discussion with your AD guys, if in fact AD is a factor here. Even if the users aren't *consciously* looking up AD-related names, if the AD domain is in the Suffix Search List and your users' shortname addiction is out of control, the combination of the two, along with excess NS records at the apex, can ultimately result in a lot of bogus TCP retries. Sometimes you can alleviate this with careful ordering or pruning of elements in the Suffix Search List. A lot of folks think that query logging is a drain on resources, and anyone who is serious about DNS performance would never turn it on. Those folks must not work in a large, chaotic enterprise :-) I find query logging and associated data-mining tools I've developed over the years, invaluable to track down broken and/or obsolete query traffic and eliminate it at the source. This saves me *much* more performance than the query logging itself, as well as being valuable for security forensics, incident avoidance (e.g. before I delete this name from DNS, let me check whether anyone is still looking it up) and a plethora of other useful stuff. - Kevin On 10/19/2013 9:34 PM, brett smith wrote: When all the Windows PC's are switched to our resolver, bind stops responding. rndc querylog shows queries coming thru, I changed tcp-clients from 1000 to 1 but DNS seems lagging, so we switched back to the original Windows Domain resolver. Besides increasing open files tuning, what TCP / sysctl or named.conf settings can be set to optimize / speed up DNS queries? Because it seems that Windows clients use TCP instead of UDP when looking at netstat on the server. Thanks. Brett. On Sat, Oct 19, 2013 at 3:20 AM, sth...@nethelp.no wrote: I need to build a pair DNS cache servers to support 5000+ clients ( PC's and Servers ). I have been looking for some guides on tuning BIND and the OS for Enterprise performance rather than the defaults. The version of bind is bind-9.8.2. 5000 clients is such a low number that I don't think you need to worry about tuning at all. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
Yes tuning off IPTABLES conn-tracking makes a huge difference. I also followed: https://access.redhat.com/site/solutions/304713 https://access.redhat.com/site/solutions/168483 I still see some SYN_SENT from Windows PC's on tcp port 53 on the DNS cache server. Thank You, Brett On Sun, Oct 20, 2013 at 6:27 PM, Stuart Browne stuart.bro...@ausregistry.com.au wrote: -Original Message- From: bind-users-bounces+stuart.browne=ausregistry.com...@lists.isc.org [mailto:bind-users-bounces+stuart.browne=ausregistry.com...@lists.isc.org] On Behalf Of brett smith Sent: Sunday, 20 October 2013 12:35 PM To: sth...@nethelp.no Cc: bind-users@lists.isc.org Subject: Re: Performance Tuning RHEL 5 and Bind When all the Windows PC's are switched to our resolver, bind stops responding. rndc querylog shows queries coming thru, I changed tcp-clients from 1000 to 1 but DNS seems lagging, so we switched back to the original Windows Domain resolver. Besides increasing open files tuning, what TCP / sysctl or named.conf settings can be set to optimize / speed up DNS queries? Because it seems that Windows clients use TCP instead of UDP when looking at netstat on the server. Thanks. Brett. On Sat, Oct 19, 2013 at 3:20 AM, sth...@nethelp.no wrote: I need to build a pair DNS cache servers to support 5000+ clients ( PC's and Servers ). I have been looking for some guides on tuning BIND and the OS for Enterprise performance rather than the defaults. The version of bind is bind-9.8.2. 5000 clients is such a low number that I don't think you need to worry about tuning at all. Steinar Haug, Nethelp consulting, sth...@nethelp.no If my experience with high-throughput through a redhat system is anything to go by, what you are probably hitting is the IPTables conntrack bucket limits. The simplest way to avoid this is to bypass connection tracking. You can do one of the following: - Turn off iptables (probably not a good idea) - Turn off conn-tracking and not use the state module, rewriting all rules (nasty) - Tell iptables to not conntrack for just udp/53 tcp/53 (-A -t raw -j NOTRACK -m tcp -p tcp --dport 53 ; -A -t raw -j NOTRACK -m udp -p udp --dport 53) We use the 3rd method and it works beautifully. Just ensure you're 'filter' rules don't force the use of conntrack for that traffic. See the man page for more details. Stuart ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
On Oct 22, 2013, at 8:29 PM, brett smith brett.s9...@gmail.com wrote: Yes tuning off IPTABLES conn-tracking makes a huge difference. I also followed: https://access.redhat.com/site/solutions/304713 https://access.redhat.com/site/solutions/168483 I still see some SYN_SENT from Windows PC's on tcp port 53 on the DNS cache server. You've cured the symptoms, not the illness. You really, REALLY need to figure out why your clients are doing TCP. You'll see a world of difference when you solve this part of the puzzle. AlanC -- Alan Clegg | +1-919-355-8851 | a...@clegg.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
From: Alan Clegg a...@clegg.com Fix your windows clients. You can't fix stupid. Confidentiality Notice: This electronic message and any attachments may contain confidential or privileged information, and is intended only for the individual or entity identified above as the addressee. If you are not the addressee (or the employee or agent responsible to deliver it to the addressee), or if this message has been addressed to you in error, you are hereby notified that you may not copy, forward, disclose or use any part of this message or any attachments. Please notify the sender immediately by return e-mail or telephone and delete this message from your system. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: Performance Tuning RHEL 5 and Bind
Any reason you're using RHEL5 as opposed to RHEL6 if you're building new servers? RHEL5 is very long in the tooth and will go EOL sooner than RHEL6. Since you're using a BIND package not shipped with RHEL5 there's no reason on that account not to move up to RHEL6. -Original Message- From: bind-users-bounces+jlightner=water@lists.isc.org [mailto:bind-users-bounces+jlightner=water@lists.isc.org] On Behalf Of wbr...@e1b.org Sent: Monday, October 21, 2013 9:47 AM To: bind-users@lists.isc.org Subject: Re: Performance Tuning RHEL 5 and Bind From: Alan Clegg a...@clegg.com Fix your windows clients. You can't fix stupid. Confidentiality Notice: This electronic message and any attachments may contain confidential or privileged information, and is intended only for the individual or entity identified above as the addressee. If you are not the addressee (or the employee or agent responsible to deliver it to the addressee), or if this message has been addressed to you in error, you are hereby notified that you may not copy, forward, disclose or use any part of this message or any attachments. Please notify the sender immediately by return e-mail or telephone and delete this message from your system. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users Athena(r), Created for the Cause(tm) Making a Difference in the Fight Against Breast Cancer - CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you. -- ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
On 20 October 2013 02:34, brett smith brett.s9...@gmail.com wrote: When all the Windows PC's are switched to our resolver, bind stops responding. rndc querylog shows queries coming thru, I changed tcp-clients from 1000 to 1 but DNS seems lagging, so we switched back to the original Windows Domain resolver. Besides increasing open files tuning, what TCP / sysctl or named.conf settings can be set to optimize / speed up DNS queries? Because it seems that Windows clients use TCP instead of UDP when looking at netstat on the server. It will depend on the type and size of the query (and on the configuration/structure of the network in-between) as to whether Windows uses UDP or is forced to switch to TCP. But the option you are probably looking for is recursive-clients and then pick a number. The default is 1000, so this is probably why if all of your systems are querying at once it stops responding to some of them. Other than that it's a case of how much memory, CPU. Is it a VM? if so have you reserved enough resources for it? What data is it serving? caching only? authoritative for any zones? Is query logging enabled? (this is a big performance hit as it has to write everything to disk, so your disk is going to be a bottleneck). Tuning is not something that you can be told this is what to do, there are a huge number of factors that will influence which parameters to tweak. But I'd definitely look to the recursive-clients option for starters. Steve ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
On Oct 19, 2013, at 9:34 PM, brett smith brett.s9...@gmail.com wrote: When all the Windows PC's are switched to our resolver, bind stops responding. What does stops responding mean? Any logs? rndc querylog shows queries coming thru, I changed tcp-clients from 1000 to 1 but DNS seems lagging, so we switched back to the original Windows Domain resolver. Are you really getting that many TCP based queries? If so, something is seriously broken. Besides increasing open files tuning, what TCP / sysctl or named.conf settings can be set to optimize / speed up DNS queries? Because it seems that Windows clients use TCP instead of UDP when looking at netstat on the server. Fix your windows clients. AlanC -- Alan Clegg | +1-919-355-8851 | a...@clegg.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
RE: Performance Tuning RHEL 5 and Bind
-Original Message- From: bind-users-bounces+stuart.browne=ausregistry.com...@lists.isc.org [mailto:bind-users-bounces+stuart.browne=ausregistry.com...@lists.isc.org] On Behalf Of brett smith Sent: Sunday, 20 October 2013 12:35 PM To: sth...@nethelp.no Cc: bind-users@lists.isc.org Subject: Re: Performance Tuning RHEL 5 and Bind When all the Windows PC's are switched to our resolver, bind stops responding. rndc querylog shows queries coming thru, I changed tcp-clients from 1000 to 1 but DNS seems lagging, so we switched back to the original Windows Domain resolver. Besides increasing open files tuning, what TCP / sysctl or named.conf settings can be set to optimize / speed up DNS queries? Because it seems that Windows clients use TCP instead of UDP when looking at netstat on the server. Thanks. Brett. On Sat, Oct 19, 2013 at 3:20 AM, sth...@nethelp.no wrote: I need to build a pair DNS cache servers to support 5000+ clients ( PC's and Servers ). I have been looking for some guides on tuning BIND and the OS for Enterprise performance rather than the defaults. The version of bind is bind-9.8.2. 5000 clients is such a low number that I don't think you need to worry about tuning at all. Steinar Haug, Nethelp consulting, sth...@nethelp.no If my experience with high-throughput through a redhat system is anything to go by, what you are probably hitting is the IPTables conntrack bucket limits. The simplest way to avoid this is to bypass connection tracking. You can do one of the following: - Turn off iptables (probably not a good idea) - Turn off conn-tracking and not use the state module, rewriting all rules (nasty) - Tell iptables to not conntrack for just udp/53 tcp/53 (-A -t raw -j NOTRACK -m tcp -p tcp --dport 53 ; -A -t raw -j NOTRACK -m udp -p udp --dport 53) We use the 3rd method and it works beautifully. Just ensure you're 'filter' rules don't force the use of conntrack for that traffic. See the man page for more details. Stuart ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Performance Tuning RHEL 5 and Bind
I need to build a pair DNS cache servers to support 5000+ clients ( PC's and Servers ). I have been looking for some guides on tuning BIND and the OS for Enterprise performance rather than the defaults. The version of bind is bind-9.8.2. 5000 clients is such a low number that I don't think you need to worry about tuning at all. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
