DNS migration and folks that don't play nice
Preface - The folks on the sys-admin list are talking about the migration of services from the older server to the newer server. Of course, one of the issues that's come up is DNS. This led to the following snippet: On Sat, 2006-04-08 at 09:04 -0400, wrote: Well, there's at least one easy workaround for that, aside from the obvious (shorten TTL ahead of time, to force fast propagation). Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS. I was curious - how do folks in general deal with this? While AOL can certainly constitute a large number of users, my inclination is to say hell with 'em. If they can't conform to proper netiquette, why should I be bending over backwards to support them? I was just curious to get other folks' take on this quasi-philosophical point. -- Cole Tuininga [EMAIL PROTECTED] ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
Cole Tuininga wrote: Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS. I was curious - how do folks in general deal with this? While AOL can certainly constitute a large number of users, my inclination is to say hell with 'em. If they can't conform to proper netiquette, why should I be bending over backwards to support them? I was just curious to get other folks' take on this quasi-philosophical point. I wasn't aware that AOL was screwing this up as well. However, I don't see anything that can be done about their blatant disregard for the way DNS is designed to work. Saying the hell with 'em is probably your only realistic option. -- John Abreau / Executive Director, Boston Linux Unix ICQ 28611923 / AIM abreauj / JABBER [EMAIL PROTECTED] / YAHOO abreauj Email [EMAIL PROTECTED] / WWW http://www.abreau.net / PGP-Key-ID 0xD5C7B5D9 PGP-Key-Fingerprint 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99 ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On Mon, 2006-04-10 at 10:27 -0400, John Abreau wrote: Cole Tuininga wrote: I wasn't aware that AOL was screwing this up as well. Last I was aware, AOL cached DNS entries for a minimum of two weeks, no matter what the TTL. However, I don't see anything that can be done about their blatant disregard for the way DNS is designed to work. Saying the hell with 'em is probably your only realistic option. Well, some folks take the approach that they will try to make sure services remain forwarding for at least two weeks, to accommodate this. As I try to remember to set TTL's to a low value for a while before making changes, I usually say to hell with 'em and only support the forwarding for a little longer than the TTL allows fo -- Cole Tuininga [EMAIL PROTECTED] ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On Mon, April 10, 2006 10:27 am, John Abreau wrote: I wasn't aware that AOL was screwing this up as well. However, I don't see anything that can be done about their blatant disregard for the way DNS is designed to work. There's actually one nice side-benefit I've noticed: some spammers (unsurprisingly) also violate DNS stuff, and cache the MX record for, well, a long, long time. It was kind of amusing to see spam attempts, addressed correctly, but going to a server that was no longer forwarding the e-mail -- and this went on for *months*. -Ken ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On 4/10/06, Cole Tuininga [EMAIL PROTECTED] wrote: Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS. I was curious - how do folks in general deal with this? There's nothing much you can do about Internet brain damage, so all you can do is plan for it. When it comes to service migration, there are usually things one can do to work around any TTL issues. These are a good idea even without deliberate brain damage -- accidental brain damage is common enough. For example, when it comes to migrating mail, we're going to implement a mechanism where the old system forwards mail to the new for some time after changing the MX records. We can monitor logs to see how things progress. If think DNS TTL brain damage is bad, try path MTU discovery some time... While AOL can certainly constitute a large number of users, my inclination is to say hell with 'em. Me too. Alas, I've found a large number of paying customers either use AOL themselves, or have customers who do. AOL claims their resolvers properly honor TTL (http://dns.info.aol.com/). I don't know if one should believe them or not. It may have been a past behavior thing. OTOH, AOL is big enough and incompetent enough that they might think they are doing things right but still have non-compliant resolvers. If they can't conform to proper netiquette, why should I be bending over backwards to support them? With AOL, it's usually more like bending over forwards... -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On Mon, 2006-04-10 at 10:04 -0400, Cole Tuininga wrote: Preface - The folks on the sys-admin list are talking about the migration of services from the older server to the newer server. Of course, one of the issues that's come up is DNS. This led to the following snippet: On Sat, 2006-04-08 at 09:04 -0400, wrote: Well, there's at least one easy workaround for that, aside from the obvious (shorten TTL ahead of time, to force fast propagation). Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS. I was curious - how do folks in general deal with this? (Context is HTTP and SMTP servers) Usually, I will try to run in parallel for up to 10 days. I'll also watch the logs a bit to see how quickly traffic dries up at the old site. When serving static pages, this is pretty painless. It is also fairly easy to migrate data that gets posted to a RDBMS on the old site. The last site I moved, HowsYourBaby worked quite smoothly. The old site usage dried up in a day except for 1 laggard who showed up about 5 days later. (Could not find the record now, but I think that's accurate.) I pulled off the laggard data from the old DB and reposted it to the new DB after the 10 day wait. Yes this means paying double hosting fees for a month. While AOL can certainly constitute a large number of users, my inclination is to say hell with 'em. If they can't conform to proper netiquette, why should I be bending over backwards to support them? I was just curious to get other folks' take on this quasi-philosophical point. -- Lloyd Kvam Venix Corp ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On Mon, Apr 10, 2006 at 10:04:53AM -0400, Cole Tuininga wrote: Preface - The folks on the sys-admin list are talking about the migration of services from the older server to the newer server. Of course, one of the issues that's come up is DNS. This led to the following snippet: On Sat, 2006-04-08 at 09:04 -0400, wrote: Well, there's at least one easy workaround for that, aside from the obvious (shorten TTL ahead of time, to force fast propagation). Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS. I was curious - how do folks in general deal with this? While AOL can certainly constitute a large number of users, my inclination is to say hell with 'em. If they can't conform to proper netiquette, why should I be bending over backwards to support them? I was just curious to get other folks' take on this quasi-philosophical point. Any evidence of this? I've got a friend at AOL (who knows of such things) and says they're using BIND and thus are honoring TTL. -Mark signature.asc Description: Digital signature
Re: DNS migration and folks that don't play nice
Another term in this equation is that your average AOL user is just slighter dumber than their computer -- with the power off. They're more likely to have misconfigured settings, spyware, DNS hijacking, other badware, obsolete software, etc. That sure doesn't help. -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On 4/10/06, Cole Tuininga [EMAIL PROTECTED] wrote: Preface -The folks on the sys-admin list are talking about the migration ofservices from the older server to the newer server.Of course, one ofthe issues that's come up is DNS.This led to the following snippet: On Sat, 2006-04-08 at 09:04 -0400, wrote: Well, there's at least one easy workaround for that, aside from the obvious (shorten TTL ahead of time, to force fast propagation). Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS.I was curious - how do folks in general deal with this?While AOL cancertainly constitute a large number of users, my inclination is to sayhell with 'em.If they can't conform to proper netiquette, why should I be bending over backwards to support them? Becouse your users may be using them. ;-) Best suggestion is, add the new DNS servers into the root server, so that both the old AND new servers are present. Wait for this to propogate, bring up the new servers, bring down the old, and remove the old servers entries. Doing it over a period of a few days, tends to work best. Thomas
Re: DNS migration and folks that don't play nice
Cole Tuininga wrote: On Mon, 2006-04-10 at 10:58 -0400, Mark Komarinski wrote: Any evidence of this? Nope - my knowledge is both anecdotal and quite possibly very out of date. Yes, but not recent, and not in the form of log files. I used AOL merely to indicate that there are some large organizations that have what appears to be deliberately broken DNS servers. I've got a friend at AOL (who knows of such things) and says they're using BIND and thus are honoring TTL. That explains it! Older versions of BIND had problems - they were especially vulnerable to attacks, and fell down in pathologically bad ways. It got to the point where I was restarting BIND every two days until they (ISC) started coming out with security fixes. Interesting - this does seem counter to the experience a few of my (less tech savvy) friends who make use of aol. I wonder - perhaps the aol software itself caches the lookups? I dunno. There's lots of crufty software between BIND and the resolver. And the resolver's cache could easily be scrod. I would not be surprised at all if it looked like a BIND server was operating correctly for a few zones, and not others. Add to this the fact that most BIND servers operate using UDP instead of TCP, and its easy to understand how BIND servers could become corrupt. Add to this the amount of malware on the Internet, and its surprising that things are working at all! --Bruce ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
On Mon, Apr 10, 2006 at 10:04:53AM -0400, Cole Tuininga wrote: Preface - The folks on the sys-admin list are talking about the migration of services from the older server to the newer server. Of course, one of the issues that's come up is DNS. This led to the following snippet: On Sat, 2006-04-08 at 09:04 -0400, wrote: Well, there's at least one easy workaround for that, aside from the obvious (shorten TTL ahead of time, to force fast propagation). When we change a host's IP address, we drop the TTL to 300 seconds a few days before, make the change, then raise it back up to 1 day. We don't have many AOL users, but so far haven't had any complaints from users that they can't reach the site or hit the wrong site. -Mark signature.asc Description: Digital signature
Re: DNS migration and folks that don't play nice
On 4/10/06, Mark Komarinski [EMAIL PROTECTED] wrote: When we change a host's IP address, we drop the TTL to 300 seconds a few days before, make the change, then raise it back up to 1 day. Ideally, one does a ramp down on the TTL. For example, if your TTL is set to one week normally, then one week in advance, you reduce the TTL to six days. Six days out. you, you reduce it to five. And so on. Use a little padding. I believe DJB's DNS tools have a feature that does this automagically. -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
Bruce Dawson writes: Add to this the fact that most BIND servers operate using UDP instead of TCP, and its easy to understand how BIND servers could become corrupt. How does the fact that a BIND server uses TCP instead of UDP make it more or less secure? (I don't know; this is why I ask) Thanks, --kevin -- GnuPG ID: B280F24E And the madness of the crowd alumni.unh.edu!kdc Is an epileptic fit -- Tom Waits ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
Kevin D. Clark wrote: Bruce Dawson writes: Add to this the fact that most BIND servers operate using UDP instead of TCP, and its easy to understand how BIND servers could become corrupt. How does the fact that a BIND server uses TCP instead of UDP make it more or less secure? Its more a reliability than a security issue. UDP is more suseptible to DOS attacks than TCP. Its also easier to spoof (largely because its simpler than TCP). Keep in mind that TCP has packet counts, checksums, ... UDP has none of that. --Bruce ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
Cole Tuininga wrote: On Mon, 2006-04-10 at 10:27 -0400, John Abreau wrote: Cole Tuininga wrote: I wasn't aware that AOL was screwing this up as well. Last I was aware, AOL cached DNS entries for a minimum of two weeks, no matter what the TTL. However, I don't see anything that can be done about their blatant disregard for the way DNS is designed to work. Saying the hell with 'em is probably your only realistic option. Well, some folks take the approach that they will try to make sure services remain forwarding for at least two weeks, to accommodate this. As I try to remember to set TTL's to a low value for a while before making changes, I usually say to hell with 'em and only support the forwarding for a little longer than the TTL allows fo If you're doing that for an enterprise, sure; but does GNHLUG have the resources and spare machine to do that for the server migration? -- John Abreau / Executive Director, Boston Linux Unix ICQ 28611923 / AIM abreauj / JABBER [EMAIL PROTECTED] / YAHOO abreauj Email [EMAIL PROTECTED] / WWW http://www.abreau.net / PGP-Key-ID 0xD5C7B5D9 PGP-Key-Fingerprint 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99 ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
Cole Tuininga writes: Preface - The folks on the sys-admin list are talking about the migration of services from the older server to the newer server. Of course, one of the issues that's come up is DNS. This led to the following snippet: On Sat, 2006-04-08 at 09:04 -0400, wrote: Well, there's at least one easy workaround for that, aside from the obvious (shorten TTL ahead of time, to force fast propagation). Unfortunately, shortening the TTL doesn't work for clients (like AOL) that cache/maintain their own DNS. I was curious - how do folks in general deal with this? While AOL can certainly constitute a large number of users, my inclination is to say hell with 'em. If they can't conform to proper netiquette, why should I be bending over backwards to support them? I was just curious to get other folks' take on this quasi-philosophical point. For HTTP you can create temporary A/PTRs that have never existed then use a 302 to redirect from old to new. For example: old server has www.example.com that responds with a 302 redirecting to www2.example.com new server hosts both www and www2 with the same content. That way people with and old cache will request a new lookup for www2 (which is new and never had the old address). This of course means you need to keep the www2 name around indefinately because it could end up in people's bookmarks/links. If bandwidth isn't an issue for the short term, the better solution is to NAT requests going to the old server to the new server. Use both SNAT and DNAT in iptables to redirect important UDP/TCP ports on the old server to the new server. -- Dave ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
Bruce Dawson [EMAIL PROTECTED] writes: That explains it! Older versions of BIND had problems - they were especially vulnerable to attacks, and fell down in pathologically bad ways. It got to the point where I was restarting BIND every two days until they (ISC) started coming out with security fixes. [...] I would not be surprised at all if it looked like a BIND server was operating correctly for a few zones, and not others. Add to this the fact that most BIND servers operate using UDP instead of TCP, and its easy to understand how BIND servers could become corrupt. Add to this the amount of malware on the Internet, and its surprising that things are working at all! We just migrated to a new BIND server and finally retired our very old and tired NetBSD machine. The NetBSD machine was 5+ years old, and was already tired when I inherited 2.5 years ago. As people have probably suspected for a while, the network I currently manage is, ahm, a little on the irregular side of things :) For Directory Services, we run Hesiod, which is essentially nothing more than using DNS TXT and CNAME records to wrap around your /etc/passwd file and serve them up using a DNS server. It's quite lightweight, and very fast. However, our primary DNS server was our slave Hesiod server, and vice versa. For some reason, whenever we updated the records on the Hesiod server we had to actually kill off the named running on the primary dns server for it to update it's copy of the hesiod domain. I have no idea why, but nothing else would update the primary servers cache of the domain except a hard restart of named. The only (ONLY he says, as if this is a *small* thing when discussing BIND :) was that the primary was running BIND9 and the Hesiod servers are running BIND8. This really *shouldn't* matter, and indeed, the new server we're running as our primary is also running BIND9 with nothing changing on the Hesiod servers, and the update just works with no restart necessary on the new BIND9 server. So, yeah, BIND can be wacky at times :) Oh, an as far as the original question goes, I usually just shorten the TTLs leading up to the event, make the switch, and wait for the rest of the world to catch up. I've never bothered to maintain forwarders for any length of time, but then again, I've only had these events happen 3 or 4 times over the past decade and it's just never been a problem. If I were running a big site where I might miss one in 2 billion e-mails comming in, or a trading site or something, I might be more cautious :) -- Seeya, Paul ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
Re: DNS migration and folks that don't play nice
[EMAIL PROTECTED] (Kevin D. Clark) writes: Bruce Dawson writes: Add to this the fact that most BIND servers operate using UDP instead of TCP, and its easy to understand how BIND servers could become corrupt. How does the fact that a BIND server uses TCP instead of UDP make it more or less secure? (I don't know; this is why I ask) I think it's more a reliability thing than security (though one could argue reliability is part of good security...) If you're name servers are receiving updates via UDP, it's far easier to drop updates in the zone transfer since UDP is lacking everything required to guarantee a complete transaction. Moving your zone transfers over to a TCP connection do a lot more to guarantee the entire update completes correctly. Note, though, usually, BIND is configured for zone transfers to occur over TCP, not the average resolver query. That still happen over UDP as far as I know. -- Seeya, Paul ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
