Re: Optimising rndc reload times on a slave server with 50,000 zones
Running off SSDs has also proved to help startup/reload times in our usage. Dan Durrer No-IP On Mar 2, 2011, at 5:32 AM, david klein wrote: > One other thing: on the filesystem in which reside directories that > house the zone files, set the mount option "noatime". This will > improve the performance of re-reading the zone files because it will > take out the necessity of updating a time-stamp for each read. > > > -DTK > > > On Mon, Feb 28, 2011 at 7:34 AM, david klein wrote: >> 5 files in a single directory will make difficult for any >> filesystem. I would recommend breaking that out into groups of less >> than 1 per directory. For better performance, separate them onto >> directories that are on different spindles; the parallelization of >> seek (and with thousands of small files that can each be read in one >> or two reads, your disks will spend a lot of this time seeking) should >> show noticeable performance improvement. >> >> Do only some of the zones update at any given 15 minute cycle? If so, >> you may show an even bigger improvement by only reloading those that >> will have changed. >> >> >> >> On Sat, Feb 26, 2011 at 8:56 PM, Dennis Perisa >> wrote: >>> Hi folks, >>> I'm looking for suggestions to substantially improve reload times on a slave >>> that is serving 50,000 zones (mostly customer zones). >>> 'rndc reload' is being executed on the slave every 15 minutes. Due to the >>> large number of zones to trawl through, the reload process is causing >>> intermittent outages and/or significant delays to zone transfers. >>> Here are some ideas I have: >>> - use rndc reconfig instead >>> - separate zone files into separate dirs to improve O/S performance >>> (currently, all zone files are in a single dir) >>> Are these viable options? Any other thoughts/suggestions? >>> This is expected to be a short-term fix while we consider brute force >>> approach of throwing more cpu/mem/IO at this. >>> DP >>> >>> ___ >>> bind-users mailing list >>> bind-users@lists.isc.org >>> https://lists.isc.org/mailman/listinfo/bind-users >>> >> >> >> >> -- >> >> david t. klein >> >> Cisco Certified Network Associate (CSCO11281885) >> Linux Professional Institute Certification (LPI000165615) >> Redhat Certified Engineer (805009745938860) >> >> Quis custodiet ipsos custodes? >> > > > > -- > > david t. klein > > Cisco Certified Network Associate (CSCO11281885) > Linux Professional Institute Certification (LPI000165615) > Redhat Certified Engineer (805009745938860) > > Quis custodiet ipsos custodes? > ___ > bind-users mailing list > bind-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/bind-users ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Optimising rndc reload times on a slave server with 50,000 zones
One other thing: on the filesystem in which reside directories that house the zone files, set the mount option "noatime". This will improve the performance of re-reading the zone files because it will take out the necessity of updating a time-stamp for each read. -DTK On Mon, Feb 28, 2011 at 7:34 AM, david klein wrote: > 5 files in a single directory will make difficult for any > filesystem. I would recommend breaking that out into groups of less > than 1 per directory. For better performance, separate them onto > directories that are on different spindles; the parallelization of > seek (and with thousands of small files that can each be read in one > or two reads, your disks will spend a lot of this time seeking) should > show noticeable performance improvement. > > Do only some of the zones update at any given 15 minute cycle? If so, > you may show an even bigger improvement by only reloading those that > will have changed. > > > > On Sat, Feb 26, 2011 at 8:56 PM, Dennis Perisa > wrote: >> Hi folks, >> I'm looking for suggestions to substantially improve reload times on a slave >> that is serving 50,000 zones (mostly customer zones). >> 'rndc reload' is being executed on the slave every 15 minutes. Due to the >> large number of zones to trawl through, the reload process is causing >> intermittent outages and/or significant delays to zone transfers. >> Here are some ideas I have: >> - use rndc reconfig instead >> - separate zone files into separate dirs to improve O/S performance >> (currently, all zone files are in a single dir) >> Are these viable options? Any other thoughts/suggestions? >> This is expected to be a short-term fix while we consider brute force >> approach of throwing more cpu/mem/IO at this. >> DP >> >> ___ >> bind-users mailing list >> bind-users@lists.isc.org >> https://lists.isc.org/mailman/listinfo/bind-users >> > > > > -- > > david t. klein > > Cisco Certified Network Associate (CSCO11281885) > Linux Professional Institute Certification (LPI000165615) > Redhat Certified Engineer (805009745938860) > > Quis custodiet ipsos custodes? > -- david t. klein Cisco Certified Network Associate (CSCO11281885) Linux Professional Institute Certification (LPI000165615) Redhat Certified Engineer (805009745938860) Quis custodiet ipsos custodes? ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Optimising rndc reload times on a slave server with 50,000 zones
On 2/27/2011 1:15 AM, Dennis Perisa wrote: > Thanks Doug. Yes, helps a lot. And yes, this is to handle adding new > zones. Look into BIND 9.7.2 or newer and the "rndc addzone" capabilities. Solves the problem without needing to reload/restart/reconifg at all. AlanC signature.asc Description: OpenPGP digital signature ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Optimising rndc reload times on a slave server with 50,000 zones
5 files in a single directory will make difficult for any filesystem. I would recommend breaking that out into groups of less than 1 per directory. For better performance, separate them onto directories that are on different spindles; the parallelization of seek (and with thousands of small files that can each be read in one or two reads, your disks will spend a lot of this time seeking) should show noticeable performance improvement. Do only some of the zones update at any given 15 minute cycle? If so, you may show an even bigger improvement by only reloading those that will have changed. On Sat, Feb 26, 2011 at 8:56 PM, Dennis Perisa wrote: > Hi folks, > I'm looking for suggestions to substantially improve reload times on a slave > that is serving 50,000 zones (mostly customer zones). > 'rndc reload' is being executed on the slave every 15 minutes. Due to the > large number of zones to trawl through, the reload process is causing > intermittent outages and/or significant delays to zone transfers. > Here are some ideas I have: > - use rndc reconfig instead > - separate zone files into separate dirs to improve O/S performance > (currently, all zone files are in a single dir) > Are these viable options? Any other thoughts/suggestions? > This is expected to be a short-term fix while we consider brute force > approach of throwing more cpu/mem/IO at this. > DP > > ___ > bind-users mailing list > bind-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/bind-users > -- david t. klein Cisco Certified Network Associate (CSCO11281885) Linux Professional Institute Certification (LPI000165615) Redhat Certified Engineer (805009745938860) Quis custodiet ipsos custodes? ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Optimising rndc reload times on a slave server with 50,000 zones
Thanks Doug. Yes, helps a lot. And yes, this is to handle adding new zones. Glad I'm on the right track :) I should point out that those ideas came from trawling this and many other forums! I should also point out that when I said "short-term fix", I meant these were changes we could implement relatively quickly compared to upgrading hardware. These fixes would certainly remain in place for the long term. 600,000 zones!! Can I ask what type of hardware you were using? We are unfortunately running on old h/w and o/s - hp dl380 g4 with FreeBSD 6. So any other tips for optimisation would be greatly appreciated. DP On Sun, Feb 27, 2011 at 3:22 PM, Doug Barton wrote: > On 02/26/2011 18:56, Dennis Perisa wrote: > >> Hi folks, >> >> I'm looking for suggestions to substantially improve reload times on a >> slave that is serving 50,000 zones (mostly customer zones). >> >> 'rndc reload' is being executed on the slave every 15 minutes. >> > > Yeah, don't do that. :) Is this being done to handle updates to customer > zones? If so, you're much better off to reload them individually (rndc > reload zonename), or better yet use dynamic updates. If you're doing this > for other reasons you need to expand. > > > Due to >> the large number of zones to trawl through, the reload process is >> causing intermittent outages and/or significant delays to zone transfers. >> > > Familiar problem. > > > Here are some ideas I have: >> - use rndc reconfig instead >> > > This sounds like you're trying to handle _adding_ new customer zones. In > that case, yes you're much better off with reconfig. > > > - separate zone files into separate dirs to improve O/S performance >> (currently, all zone files are in a single dir) >> > > Depending on your OS this can help quite a bit. 10k files in one directory > is the point where you can start to see problems, so 50k is likely well past > this point. I had a structure with one level based on the first character of > the domain, and the next level based on the first 2 characters. That way you > should be able to keep all the directories down to less than 5k files (due > to uneven distribution of first characters in domain names). > > > Are these viable options? Any other thoughts/suggestions? >> > > I did all 3 of these things in the past to handle over 600,000 zones, so > you're definitely on the right track. > > > This is expected to be a short-term fix while we consider brute force >> approach of throwing more cpu/mem/IO at this. >> > > Depending on how much you plan to grow going forward you probably shouldn't > consider these short-term steps. > > > hth, > > Doug > > -- > >Nothin' ever doesn't change, but nothin' changes much. >-- OK Go > >Breadth of IT experience, and depth of knowledge in the DNS. >Yours for the right price. :) http://SupersetSolutions.com/ > > ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Optimising rndc reload times on a slave server with 50,000 zones
On 02/26/2011 18:56, Dennis Perisa wrote: Hi folks, I'm looking for suggestions to substantially improve reload times on a slave that is serving 50,000 zones (mostly customer zones). 'rndc reload' is being executed on the slave every 15 minutes. Yeah, don't do that. :) Is this being done to handle updates to customer zones? If so, you're much better off to reload them individually (rndc reload zonename), or better yet use dynamic updates. If you're doing this for other reasons you need to expand. Due to the large number of zones to trawl through, the reload process is causing intermittent outages and/or significant delays to zone transfers. Familiar problem. Here are some ideas I have: - use rndc reconfig instead This sounds like you're trying to handle _adding_ new customer zones. In that case, yes you're much better off with reconfig. - separate zone files into separate dirs to improve O/S performance (currently, all zone files are in a single dir) Depending on your OS this can help quite a bit. 10k files in one directory is the point where you can start to see problems, so 50k is likely well past this point. I had a structure with one level based on the first character of the domain, and the next level based on the first 2 characters. That way you should be able to keep all the directories down to less than 5k files (due to uneven distribution of first characters in domain names). Are these viable options? Any other thoughts/suggestions? I did all 3 of these things in the past to handle over 600,000 zones, so you're definitely on the right track. This is expected to be a short-term fix while we consider brute force approach of throwing more cpu/mem/IO at this. Depending on how much you plan to grow going forward you probably shouldn't consider these short-term steps. hth, Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Optimising rndc reload times on a slave server with 50,000 zones
Hi folks, I'm looking for suggestions to substantially improve reload times on a slave that is serving 50,000 zones (mostly customer zones). 'rndc reload' is being executed on the slave every 15 minutes. Due to the large number of zones to trawl through, the reload process is causing intermittent outages and/or significant delays to zone transfers. Here are some ideas I have: - use rndc reconfig instead - separate zone files into separate dirs to improve O/S performance (currently, all zone files are in a single dir) Are these viable options? Any other thoughts/suggestions? This is expected to be a short-term fix while we consider brute force approach of throwing more cpu/mem/IO at this. DP ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
