Re: how to relocate servers transparently
Rhesa Rozendaal wrote: So here is what we'll do: - Lower the ttl on all zones three days before the move No, this isn't right. You must lower the TTL time at a bare minimum 2 * (Current TTL) ahead of time. Why? Because nameservers out in the real world will not even query your nameservers again until the TTL has expired, meaning that if you change it today, the FIRST time another nameserver that has already cached your records will ask for it again is after the *current* TTL expires. Now take the case where one nameserver is a forwarder for another (rare, but there are environments where it's needed) and the one behind the forwarder could take up to 2 * TTL to come ask for new information. Hopefully no one out there you need to hit your site is dumb enough to have a triple-forwarded DNS resolver environment. - Move all the boxes except for one dns server Better to duplicate the boxes or move half the boxes - but it really depends on how your farm is set up. If you have a true N+1 server farm environment where you have at least one spare box worth of processing power on the front-end and multiple database servers, the only problem you have between the sites is keeping the databases synchronized. A number of possible solutions can help you if you can't afford this level of redundancy... bring a webserver up on the OLD addresses after the new addresses are published in DNS and the servers are answering on the new addresses with a URL link/redirect to a NEW NAME (i.e. www2.whatever.com) and make sure the new servers will answer properly on both the new name and the old name. Why? When someone hits the old IP from cached information in their DNS server, you can send them to a brand new name which will force their DNS server to look up the record... because it's never gotten that one before. - Set both the dns servers to serve the new IP addresses Good. But you'll still have traffic going to the old addresses no matter what you do. - Move the old dns server once it no longer receives queries Good but you need to leave behind a web server too. I think that will make the move as transparent as possible. Nope. But I guarantee that it *is* possible to make a completely seamless move where no one gets lost getting to your site and you can prove it by watching the logs on the left-behind webserver (or a new box serving a static page) until no more hits go to it. I still have an uneasy feeling about dns caches out there that may keep serving the old ip addresses to their users _without_ ever consulting our dns servers. But I guess I could use a http proxy on the remaining dns box to forward http traffic for a while, which would take care of that part. The other protocols are less important (or visible), and more likely to work correctly anyway. It's not the DNS server's fault... it's your original TTL time - they're just doing what they were told to do like all computers do. Thanks again guys, I may become an experienced ISP after all :-) You'll get it... you're really close. Your idea for the proxy is better than my one above for the full blown web server... I'm just relating how I've seen it done in large farms. Nate Duehr, [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
On June 18, 2004 12:49 am, Nate Duehr wrote: No, this isn't right. You must lower the TTL time at a bare minimum 2 * (Current TTL) ahead of time. Why? Because nameservers out in the real world will not even query your nameservers again until the TTL has expired, meaning that if you change it today, the FIRST time another nameserver that has already cached your records will ask for it again is after the *current* TTL expires. Now take the case where one nameserver is a forwarder for another (rare, but there are environments where it's needed) and the one behind the forwarder could take up to 2 * TTL to come ask for new information. Can you explain that a little further? If my nameserver caches a record with TTL 86400, and someone asks for it again an hour later I hand them the record from my cache using TTL 82800 (not 86400). This is certainly what bind does, if other caching nameservers do it differently then it's a bug IMHO. I would be very surprised if it is different when DNS queries are being forward from one DNS server to another. Or did you mean something else? -- Fraser Campbell [EMAIL PROTECTED] http://www.wehave.net/ Georgetown, Ontario, Canada Debian GNU/Linux
Re: how to relocate servers transparently
On Jun 20, 2004, at 9:19 AM, Fraser Campbell wrote: On June 18, 2004 12:49 am, Nate Duehr wrote: No, this isn't right. You must lower the TTL time at a bare minimum 2 * (Current TTL) ahead of time. Why? Because nameservers out in the real world will not even query your nameservers again until the TTL has expired, meaning that if you change it today, the FIRST time another nameserver that has already cached your records will ask for it again is after the *current* TTL expires. Now take the case where one nameserver is a forwarder for another (rare, but there are environments where it's needed) and the one behind the forwarder could take up to 2 * TTL to come ask for new information. Can you explain that a little further? If my nameserver caches a record with TTL 86400, and someone asks for it again an hour later I hand them the record from my cache using TTL 82800 (not 86400). This is certainly what bind does, if other caching nameservers do it differently then it's a bug IMHO. No that's correct, but if you look at the descriptions originally given the posters were saying that as soon as the main nameserver is changed the changes show up at the nameservers that have cached the records. That's not true. Here's an example: Let's say the original poster changed that record you are talking about above... your nameserver would continue to hand out the old record until the TTL expired. Now let's say you're talking about a large company where they have implemented DNS forwarders or an internal and external environment where the internal DNS server forwards queries to a specific server or group of servers in their DMZ. Now that far internal server will have its own cache also. So the TTL has to expire on the internal server, the machine in the DMZ and only THEN will it ever query the master again. Of course, the way around this is to get the clients to go to a completely new name that never existed before -- then no one has it cached and they *must* come to the authoritative server(s). I would be very surprised if it is different when DNS queries are being forward from one DNS server to another. Or did you mean something else? I was specifically speaking about the forwarders configuration in named.conf. It is used (rarely, but is used) by some large organizations for the purposes mentioned above. So you're correct, in most cases it won't happen -- but there are people out there with some very deep DNS server cascading. Nate Duehr, [EMAIL PROTECTED]
Re: how to relocate servers transparently
That's right for the case of bind using another bind as forwarder! I mean asking the first bind or asking the forwarder at the same time would result in the same TTL. Don't know about other DNS server types. Christian - Original Message - From: Fraser Campbell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, June 20, 2004 5:19 PM Subject: Re: how to relocate servers transparently ... Can you explain that a little further? If my nameserver caches a record with TTL 86400, and someone asks for it again an hour later I hand them the record from my cache using TTL 82800 (not 86400). This is certainly what bind does, if other caching nameservers do it differently then it's a bug IMHO. I would be very surprised if it is different when DNS queries are being forward from one DNS server to another. Or did you mean something else? -- Fraser Campbell [EMAIL PROTECTED] http://www.wehave.net/ Georgetown, Ontario, Canada Debian GNU/Linux -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
Rhesa Rozendaal wrote: So here is what we'll do: - Lower the ttl on all zones three days before the move No, this isn't right. You must lower the TTL time at a bare minimum 2 * (Current TTL) ahead of time. Why? Because nameservers out in the real world will not even query your nameservers again until the TTL has expired, meaning that if you change it today, the FIRST time another nameserver that has already cached your records will ask for it again is after the *current* TTL expires. Now take the case where one nameserver is a forwarder for another (rare, but there are environments where it's needed) and the one behind the forwarder could take up to 2 * TTL to come ask for new information. Hopefully no one out there you need to hit your site is dumb enough to have a triple-forwarded DNS resolver environment. - Move all the boxes except for one dns server Better to duplicate the boxes or move half the boxes - but it really depends on how your farm is set up. If you have a true N+1 server farm environment where you have at least one spare box worth of processing power on the front-end and multiple database servers, the only problem you have between the sites is keeping the databases synchronized. A number of possible solutions can help you if you can't afford this level of redundancy... bring a webserver up on the OLD addresses after the new addresses are published in DNS and the servers are answering on the new addresses with a URL link/redirect to a NEW NAME (i.e. www2.whatever.com) and make sure the new servers will answer properly on both the new name and the old name. Why? When someone hits the old IP from cached information in their DNS server, you can send them to a brand new name which will force their DNS server to look up the record... because it's never gotten that one before. - Set both the dns servers to serve the new IP addresses Good. But you'll still have traffic going to the old addresses no matter what you do. - Move the old dns server once it no longer receives queries Good but you need to leave behind a web server too. I think that will make the move as transparent as possible. Nope. But I guarantee that it *is* possible to make a completely seamless move where no one gets lost getting to your site and you can prove it by watching the logs on the left-behind webserver (or a new box serving a static page) until no more hits go to it. I still have an uneasy feeling about dns caches out there that may keep serving the old ip addresses to their users _without_ ever consulting our dns servers. But I guess I could use a http proxy on the remaining dns box to forward http traffic for a while, which would take care of that part. The other protocols are less important (or visible), and more likely to work correctly anyway. It's not the DNS server's fault... it's your original TTL time - they're just doing what they were told to do like all computers do. Thanks again guys, I may become an experienced ISP after all :-) You'll get it... you're really close. Your idea for the proxy is better than my one above for the full blown web server... I'm just relating how I've seen it done in large farms. Nate Duehr, [EMAIL PROTECTED]
Re: how to relocate servers transparently
On June 18, 2004 12:49 am, Nate Duehr wrote: No, this isn't right. You must lower the TTL time at a bare minimum 2 * (Current TTL) ahead of time. Why? Because nameservers out in the real world will not even query your nameservers again until the TTL has expired, meaning that if you change it today, the FIRST time another nameserver that has already cached your records will ask for it again is after the *current* TTL expires. Now take the case where one nameserver is a forwarder for another (rare, but there are environments where it's needed) and the one behind the forwarder could take up to 2 * TTL to come ask for new information. Can you explain that a little further? If my nameserver caches a record with TTL 86400, and someone asks for it again an hour later I hand them the record from my cache using TTL 82800 (not 86400). This is certainly what bind does, if other caching nameservers do it differently then it's a bug IMHO. I would be very surprised if it is different when DNS queries are being forward from one DNS server to another. Or did you mean something else? -- Fraser Campbell [EMAIL PROTECTED] http://www.wehave.net/ Georgetown, Ontario, Canada Debian GNU/Linux
Re: how to relocate servers transparently
On Jun 20, 2004, at 9:19 AM, Fraser Campbell wrote: On June 18, 2004 12:49 am, Nate Duehr wrote: No, this isn't right. You must lower the TTL time at a bare minimum 2 * (Current TTL) ahead of time. Why? Because nameservers out in the real world will not even query your nameservers again until the TTL has expired, meaning that if you change it today, the FIRST time another nameserver that has already cached your records will ask for it again is after the *current* TTL expires. Now take the case where one nameserver is a forwarder for another (rare, but there are environments where it's needed) and the one behind the forwarder could take up to 2 * TTL to come ask for new information. Can you explain that a little further? If my nameserver caches a record with TTL 86400, and someone asks for it again an hour later I hand them the record from my cache using TTL 82800 (not 86400). This is certainly what bind does, if other caching nameservers do it differently then it's a bug IMHO. No that's correct, but if you look at the descriptions originally given the posters were saying that as soon as the main nameserver is changed the changes show up at the nameservers that have cached the records. That's not true. Here's an example: Let's say the original poster changed that record you are talking about above... your nameserver would continue to hand out the old record until the TTL expired. Now let's say you're talking about a large company where they have implemented DNS forwarders or an internal and external environment where the internal DNS server forwards queries to a specific server or group of servers in their DMZ. Now that far internal server will have its own cache also. So the TTL has to expire on the internal server, the machine in the DMZ and only THEN will it ever query the master again. Of course, the way around this is to get the clients to go to a completely new name that never existed before -- then no one has it cached and they *must* come to the authoritative server(s). I would be very surprised if it is different when DNS queries are being forward from one DNS server to another. Or did you mean something else? I was specifically speaking about the forwarders configuration in named.conf. It is used (rarely, but is used) by some large organizations for the purposes mentioned above. So you're correct, in most cases it won't happen -- but there are people out there with some very deep DNS server cascading. Nate Duehr, [EMAIL PROTECTED]
Re: how to relocate servers transparently
That's right for the case of bind using another bind as forwarder! I mean asking the first bind or asking the forwarder at the same time would result in the same TTL. Don't know about other DNS server types. Christian - Original Message - From: Fraser Campbell [EMAIL PROTECTED] To: debian-isp@lists.debian.org Sent: Sunday, June 20, 2004 5:19 PM Subject: Re: how to relocate servers transparently ... Can you explain that a little further? If my nameserver caches a record with TTL 86400, and someone asks for it again an hour later I hand them the record from my cache using TTL 82800 (not 86400). This is certainly what bind does, if other caching nameservers do it differently then it's a bug IMHO. I would be very surprised if it is different when DNS queries are being forward from one DNS server to another. Or did you mean something else? -- Fraser Campbell [EMAIL PROTECTED] http://www.wehave.net/ Georgetown, Ontario, Canada Debian GNU/Linux
Re: how to relocate servers transparently
man, 14.06.2004 kl. 17.22 skrev Andreas John: Hello Adrian! NAT is only working with IPs directly connected to your ethernet card. or NAT-ed IPs not arbitrary ones. Or? rgds, j. As NAT is done on the IP-layer, iptables doesn't care about ethernet and other (MAC layer) stuff, so you can NAT any IP as long as it is routeable. -- Frode Haugsgjerd [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
man, 14.06.2004 kl. 17.22 skrev Andreas John: Hello Adrian! NAT is only working with IPs directly connected to your ethernet card. or NAT-ed IPs not arbitrary ones. Or? rgds, j. As NAT is done on the IP-layer, iptables doesn't care about ethernet and other (MAC layer) stuff, so you can NAT any IP as long as it is routeable. -- Frode Haugsgjerd [EMAIL PROTECTED]
Re: how to relocate servers transparently
The biggest problem you will have is with the DNS. Set 1 of the DNS servers to the new IP, and keep 1 behind. Make sure the TTL is low... very low. Then, make sure the new DNS server on the new IP address is up and running with the old DNS server on the old IP (if possible), so at all times there is at least 1 DNS server running and active. Then do the switch. This way should help you minimize downtime. Jas - Original Message - From: Rhesa Rozendaal [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, 14 June, 2004 5:36 PM Subject: how to relocate servers transparently Hello all, I'm looking for some practical advice on moving servers from one colocation to another. In a couple of weeks, we need to move our servers to a different colocation, which means all the ip addresses will change. The servers are running regular stuff: web, mail, ftp, and two dns servers. In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues for many users. Beforehand we did lower the ttl on all the domains prior to the move, but many dns servers seemed to ignore that. On top of that, we moved both our dns servers at the same time, which was a big mistake too. So, what I'd like to hear from you is practical advice on how to avoid connection problems after the move is complete. Will it be enough to keep 1 dns server behind? I'm afraid it won't be, given the dns caching problem mentioned above. Is there a way to have that 1 dns server act as a proxy or port forwarder in some way? Can that be done between two different class A networks? Btw, the servers are running debian stable (woody). Thanks in advance, Rhesa Rozendaal -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
Rhesa Rozendaal wrote: In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues This is most often due to the old authoritive servers continuing to serve the old zone details. When an A record is refreshed, the TTL for SOA/NS rr's also refreshes, therefore the NS information 'seems' to never be out of date. Some DNS caches will continue querying the old servers due to the fact that those NS records have not expired. for many users. Beforehand we did lower the ttl on all the domains prior to the move, but many dns servers seemed to ignore that. On top of that, we moved both our dns servers at the same time, which was a big mistake too. When moving a site to new IP's/DNS servers I performed the following: Create all accounts on the new box, and copy all the files over. Setup the DNS servers to issue the new zone details. At the same time, configure the OLD servers to serve the new zone data. When the old servers are queried, they will serve the new zone data, so when an A record is refreshed, the SOA/NS records will be that of the new servers. You can then change the delegation for all domains to the new DNS servers. I guess the trick is to keep both DNS servers going at the same time for a few days, but ensure that they are all serving the NEW zone details for all domains. This would be the 'correct' way to change delegation and will also avoid 'server lock' (as ISC refer to it) as mentioned above with the NS records refreshing. So, what I'd like to hear from you is practical advice on how to avoid connection problems after the move is complete. Will it be enough to keep 1 dns server behind? I'm afraid it won't be, given the dns caching problem mentioned above. Is there a way to have that 1 dns server act as a proxy or port forwarder in some way? Can that be done between two different class A networks? As above, as long as both new and old servers are serving the same (new) zone details, there shouldnt be a problem. Brad -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
Jason, Brad, thanks for the reassurance. Rhesa Rozendaal wrote: In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues This is most often due to the old authoritive servers continuing to serve the old zone details. When an A record is refreshed, the TTL for SOA/NS rr's also refreshes, therefore the NS information 'seems' to never be out of date. Some DNS caches will continue querying the old servers due to the fact that those NS records have not expired. That last bit explains our problems exactly. We did get a new temporary dns server up on the old ip addresses, and that helped a lot. When moving a site to new IP's/DNS servers I performed the following: Create all accounts on the new box, and copy all the files over. Setup the DNS servers to issue the new zone details. At the same time, configure the OLD servers to serve the new zone data. When the old servers are queried, they will serve the new zone data, so when an A record is refreshed, the SOA/NS records will be that of the new servers. We are going to physically move our boxes, but for the dns the process will amount to the same thing. So here is what we'll do: - Lower the ttl on all zones three days before the move - Move all the boxes except for one dns server - Set both the dns servers to serve the new IP addresses - Move the old dns server once it no longer receives queries I think that will make the move as transparent as possible. I still have an uneasy feeling about dns caches out there that may keep serving the old ip addresses to their users _without_ ever consulting our dns servers. But I guess I could use a http proxy on the remaining dns box to forward http traffic for a while, which would take care of that part. The other protocols are less important (or visible), and more likely to work correctly anyway. Thanks again guys, I may become an experienced ISP after all :-) Rhesa -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
I still have an uneasy feeling about dns caches out there that may keep serving the old ip addresses to their users _without_ ever consulting our dns servers. But I guess I could use a http proxy on the remaining dns box to forward http traffic for a while, which would take care of that part. The other protocols are less important (or visible), and more likely to work correctly anyway. As long as all contactable DNS servers have the new authoritive information, there should not be any issues... Brad -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
Hello! The DNS Trick should do it. Instead of internal E-Mail Forwarding of ricochet E-Mails I think about some kind of port-based forwarding. By random I ran accross rinetd package. apt-cache show rinetd Package: rinetd Description: Internet TCP redirection server rinetd redirects TCP connections from one IP address and port to another, with basic IP-based access control. . rinetd is a single-process server which handles any number of connections to the address/port pairs specified in the file /etc/rinetd.conf. Since rinetd runs as a single process using nonblocking I/O, it is able to redirect a large number of connections without a severe impact on the machine. This makes it practical to run services on machines inside an IP masquerading firewall. Does anyone use that already? Port 25 forwarding? Does ist forward to arbitrary adresses unlike NAT-forward which does only forward to NAT adresses? Currently we don't move servers, but time will come and I want to be prepared. :-) Rgds, j. Brad wrote: Rhesa Rozendaal wrote: In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues This is most often due to the old authoritive servers continuing to serve the old zone details. When an A record is refreshed, the TTL for SOA/NS rr's also refreshes, therefore the NS information 'seems' to never be out of date. Some DNS caches will continue querying the old servers due to the fact that those NS records have not expired. for many users. Beforehand we did lower the ttl on all the domains prior to the move, but many dns servers seemed to ignore that. On top of that, we moved both our dns servers at the same time, which was a big mistake too. When moving a site to new IP's/DNS servers I performed the following: Create all accounts on the new box, and copy all the files over. Setup the DNS servers to issue the new zone details. At the same time, configure the OLD servers to serve the new zone data. When the old servers are queried, they will serve the new zone data, so when an A record is refreshed, the SOA/NS records will be that of the new servers. You can then change the delegation for all domains to the new DNS servers. I guess the trick is to keep both DNS servers going at the same time for a few days, but ensure that they are all serving the NEW zone details for all domains. This would be the 'correct' way to change delegation and will also avoid 'server lock' (as ISC refer to it) as mentioned above with the NS records refreshing. So, what I'd like to hear from you is practical advice on how to avoid connection problems after the move is complete. Will it be enough to keep 1 dns server behind? I'm afraid it won't be, given the dns caching problem mentioned above. Is there a way to have that 1 dns server act as a proxy or port forwarder in some way? Can that be done between two different class A networks? As above, as long as both new and old servers are serving the same (new) zone details, there shouldnt be a problem. Brad -- Andreas John net-lab GmbH Luisenstrasse 30b 63067 Offenbach Tel: +49 69 85700331 http://www.net-lab.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
I've added comments inline... Rhesa Rozendaal wrote: We are going to physically move our boxes, but for the dns the process will amount to the same thing. So here is what we'll do: - Lower the ttl on all zones three days before the move Lower the TTL on all zones OR INDIVIDUAL A RECORDS at least [OLD TTL] before the move...just worth clarifying. - Move all the boxes except for one dns server My opinion is don't move the DNS servers at the same time as the other servers, or as an alternative, make your delegation changes either after the move or more than the TLD's TTL before the move (i.e. for .com, more than two days before the move). just my $0.02, pt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
On Monday 14 June 2004 11:16, Andreas John wrote: Hello! apt-cache show rinetd Package: rinetd Yep, rinetd is a simple userspace proxy. I used it for a while, but it had a few unpleasant 'features' I didn't like (we were using dynamic DNS with it, so I expect you wouldn't encounter them) Once our 'remote points' were static IPs, I switched to using 'netcat' instead. Now we can create a quick TCP proxy by simply adding a line to inetd.conf: 12345stream tcp nowait root /usr/sbin/tcpd /usr/bin/nc 1.2.3.4 5678 So, any incoming TCP connection to the machine on port 12345 will then be proxied to remote host 1.2.3.4 on port 5678 :) (If you specify a port number in inetd.conf, you don't need to create a matching service name in /etc/services) Cheers, Gavin. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
On Monday 14 June 2004 14.59, Gavin Hamill wrote: Yep, rinetd is a simple userspace proxy. I used it for a while, but it had a few unpleasant 'features' I didn't like (we were using dynamic DNS with it, so I expect you wouldn't encounter them) Once our 'remote points' were static IPs, I switched to using 'netcat' instead. [...] This may be obvious, but not to me... is there any difference compared to using iptables DNAT? cheers -- vbi -- Beware of the FUD - know your enemies. This week * The Alexis de Toqueville Institue * http://fortytwo.ch/opinion/ pgp6hcz7Vt7zV.pgp Description: signature
Re: how to relocate servers transparently
On Monday 14 June 2004 14:57, Adrian 'Dagurashibanipal' von Bidder wrote: This may be obvious, but not to me... is there any difference compared to using iptables DNAT? At the time I was using rinetd/netcat to do this kind of work, I didn't know enough about iptables and the SNAT/DNAT pair which are required to do this kind of proxying. For lightweight traffic I often still choose this route, just out of habit :) Cheers, Gavin. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
Hello Adrian! NAT is only working with IPs directly connected to your ethernet card. or NAT-ed IPs not arbitrary ones. Or? rgds, j. Adrian 'Dagurashibanipal' von Bidder wrote: On Monday 14 June 2004 14.59, Gavin Hamill wrote: Yep, rinetd is a simple userspace proxy. I used it for a while, but it had a few unpleasant 'features' I didn't like (we were using dynamic DNS with it, so I expect you wouldn't encounter them) Once our 'remote points' were static IPs, I switched to using 'netcat' instead. [...] This may be obvious, but not to me... is there any difference compared to using iptables DNAT? cheers -- vbi -- Andreas John net-lab GmbH Luisenstrasse 30b 63067 Offenbach Tel: +49 69 85700331 http://www.net-lab.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
The biggest problem you will have is with the DNS. Set 1 of the DNS servers to the new IP, and keep 1 behind. Make sure the TTL is low... very low. Then, make sure the new DNS server on the new IP address is up and running with the old DNS server on the old IP (if possible), so at all times there is at least 1 DNS server running and active. Then do the switch. This way should help you minimize downtime. Jas - Original Message - From: Rhesa Rozendaal [EMAIL PROTECTED] To: debian-isp@lists.debian.org Sent: Monday, 14 June, 2004 5:36 PM Subject: how to relocate servers transparently Hello all, I'm looking for some practical advice on moving servers from one colocation to another. In a couple of weeks, we need to move our servers to a different colocation, which means all the ip addresses will change. The servers are running regular stuff: web, mail, ftp, and two dns servers. In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues for many users. Beforehand we did lower the ttl on all the domains prior to the move, but many dns servers seemed to ignore that. On top of that, we moved both our dns servers at the same time, which was a big mistake too. So, what I'd like to hear from you is practical advice on how to avoid connection problems after the move is complete. Will it be enough to keep 1 dns server behind? I'm afraid it won't be, given the dns caching problem mentioned above. Is there a way to have that 1 dns server act as a proxy or port forwarder in some way? Can that be done between two different class A networks? Btw, the servers are running debian stable (woody). Thanks in advance, Rhesa Rozendaal -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to relocate servers transparently
Rhesa Rozendaal wrote: In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues This is most often due to the old authoritive servers continuing to serve the old zone details. When an A record is refreshed, the TTL for SOA/NS rr's also refreshes, therefore the NS information 'seems' to never be out of date. Some DNS caches will continue querying the old servers due to the fact that those NS records have not expired. for many users. Beforehand we did lower the ttl on all the domains prior to the move, but many dns servers seemed to ignore that. On top of that, we moved both our dns servers at the same time, which was a big mistake too. When moving a site to new IP's/DNS servers I performed the following: Create all accounts on the new box, and copy all the files over. Setup the DNS servers to issue the new zone details. At the same time, configure the OLD servers to serve the new zone data. When the old servers are queried, they will serve the new zone data, so when an A record is refreshed, the SOA/NS records will be that of the new servers. You can then change the delegation for all domains to the new DNS servers. I guess the trick is to keep both DNS servers going at the same time for a few days, but ensure that they are all serving the NEW zone details for all domains. This would be the 'correct' way to change delegation and will also avoid 'server lock' (as ISC refer to it) as mentioned above with the NS records refreshing. So, what I'd like to hear from you is practical advice on how to avoid connection problems after the move is complete. Will it be enough to keep 1 dns server behind? I'm afraid it won't be, given the dns caching problem mentioned above. Is there a way to have that 1 dns server act as a proxy or port forwarder in some way? Can that be done between two different class A networks? As above, as long as both new and old servers are serving the same (new) zone details, there shouldnt be a problem. Brad
Re: how to relocate servers transparently
Jason, Brad, thanks for the reassurance. Rhesa Rozendaal wrote: In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues This is most often due to the old authoritive servers continuing to serve the old zone details. When an A record is refreshed, the TTL for SOA/NS rr's also refreshes, therefore the NS information 'seems' to never be out of date. Some DNS caches will continue querying the old servers due to the fact that those NS records have not expired. That last bit explains our problems exactly. We did get a new temporary dns server up on the old ip addresses, and that helped a lot. When moving a site to new IP's/DNS servers I performed the following: Create all accounts on the new box, and copy all the files over. Setup the DNS servers to issue the new zone details. At the same time, configure the OLD servers to serve the new zone data. When the old servers are queried, they will serve the new zone data, so when an A record is refreshed, the SOA/NS records will be that of the new servers. We are going to physically move our boxes, but for the dns the process will amount to the same thing. So here is what we'll do: - Lower the ttl on all zones three days before the move - Move all the boxes except for one dns server - Set both the dns servers to serve the new IP addresses - Move the old dns server once it no longer receives queries I think that will make the move as transparent as possible. I still have an uneasy feeling about dns caches out there that may keep serving the old ip addresses to their users _without_ ever consulting our dns servers. But I guess I could use a http proxy on the remaining dns box to forward http traffic for a while, which would take care of that part. The other protocols are less important (or visible), and more likely to work correctly anyway. Thanks again guys, I may become an experienced ISP after all :-) Rhesa
Re: how to relocate servers transparently
I still have an uneasy feeling about dns caches out there that may keep serving the old ip addresses to their users _without_ ever consulting our dns servers. But I guess I could use a http proxy on the remaining dns box to forward http traffic for a while, which would take care of that part. The other protocols are less important (or visible), and more likely to work correctly anyway. As long as all contactable DNS servers have the new authoritive information, there should not be any issues... Brad
Re: how to relocate servers transparently
Hello! The DNS Trick should do it. Instead of internal E-Mail Forwarding of ricochet E-Mails I think about some kind of port-based forwarding. By random I ran accross rinetd package. apt-cache show rinetd Package: rinetd Description: Internet TCP redirection server rinetd redirects TCP connections from one IP address and port to another, with basic IP-based access control. . rinetd is a single-process server which handles any number of connections to the address/port pairs specified in the file /etc/rinetd.conf. Since rinetd runs as a single process using nonblocking I/O, it is able to redirect a large number of connections without a severe impact on the machine. This makes it practical to run services on machines inside an IP masquerading firewall. Does anyone use that already? Port 25 forwarding? Does ist forward to arbitrary adresses unlike NAT-forward which does only forward to NAT adresses? Currently we don't move servers, but time will come and I want to be prepared. :-) Rgds, j. Brad wrote: Rhesa Rozendaal wrote: In the past I witnessed such a move, and there were a lot of problems with the DNS. As it turned out, many DNS servers out there kept caching the old ip addresses for over 3 days, causing a lot of connection issues This is most often due to the old authoritive servers continuing to serve the old zone details. When an A record is refreshed, the TTL for SOA/NS rr's also refreshes, therefore the NS information 'seems' to never be out of date. Some DNS caches will continue querying the old servers due to the fact that those NS records have not expired. for many users. Beforehand we did lower the ttl on all the domains prior to the move, but many dns servers seemed to ignore that. On top of that, we moved both our dns servers at the same time, which was a big mistake too. When moving a site to new IP's/DNS servers I performed the following: Create all accounts on the new box, and copy all the files over. Setup the DNS servers to issue the new zone details. At the same time, configure the OLD servers to serve the new zone data. When the old servers are queried, they will serve the new zone data, so when an A record is refreshed, the SOA/NS records will be that of the new servers. You can then change the delegation for all domains to the new DNS servers. I guess the trick is to keep both DNS servers going at the same time for a few days, but ensure that they are all serving the NEW zone details for all domains. This would be the 'correct' way to change delegation and will also avoid 'server lock' (as ISC refer to it) as mentioned above with the NS records refreshing. So, what I'd like to hear from you is practical advice on how to avoid connection problems after the move is complete. Will it be enough to keep 1 dns server behind? I'm afraid it won't be, given the dns caching problem mentioned above. Is there a way to have that 1 dns server act as a proxy or port forwarder in some way? Can that be done between two different class A networks? As above, as long as both new and old servers are serving the same (new) zone details, there shouldnt be a problem. Brad -- Andreas John net-lab GmbH Luisenstrasse 30b 63067 Offenbach Tel: +49 69 85700331 http://www.net-lab.net
Re: how to relocate servers transparently
I've added comments inline... Rhesa Rozendaal wrote: We are going to physically move our boxes, but for the dns the process will amount to the same thing. So here is what we'll do: - Lower the ttl on all zones three days before the move Lower the TTL on all zones OR INDIVIDUAL A RECORDS at least [OLD TTL] before the move...just worth clarifying. - Move all the boxes except for one dns server My opinion is don't move the DNS servers at the same time as the other servers, or as an alternative, make your delegation changes either after the move or more than the TLD's TTL before the move (i.e. for .com, more than two days before the move). just my $0.02, pt
Re: how to relocate servers transparently
On Monday 14 June 2004 11:16, Andreas John wrote: Hello! apt-cache show rinetd Package: rinetd Yep, rinetd is a simple userspace proxy. I used it for a while, but it had a few unpleasant 'features' I didn't like (we were using dynamic DNS with it, so I expect you wouldn't encounter them) Once our 'remote points' were static IPs, I switched to using 'netcat' instead. Now we can create a quick TCP proxy by simply adding a line to inetd.conf: 12345stream tcp nowait root /usr/sbin/tcpd /usr/bin/nc 1.2.3.4 5678 So, any incoming TCP connection to the machine on port 12345 will then be proxied to remote host 1.2.3.4 on port 5678 :) (If you specify a port number in inetd.conf, you don't need to create a matching service name in /etc/services) Cheers, Gavin.
Re: how to relocate servers transparently
On Monday 14 June 2004 14.59, Gavin Hamill wrote: Yep, rinetd is a simple userspace proxy. I used it for a while, but it had a few unpleasant 'features' I didn't like (we were using dynamic DNS with it, so I expect you wouldn't encounter them) Once our 'remote points' were static IPs, I switched to using 'netcat' instead. [...] This may be obvious, but not to me... is there any difference compared to using iptables DNAT? cheers -- vbi -- Beware of the FUD - know your enemies. This week * The Alexis de Toqueville Institue * http://fortytwo.ch/opinion/ pgp9ByZcj035n.pgp Description: signature
Re: how to relocate servers transparently
On Monday 14 June 2004 09:57, Adrian 'Dagurashibanipal' von Bidder wrote: This may be obvious, but not to me... is there any difference compared to using iptables DNAT? I believe that you'd have some problems if you used DNAT. Think of what happens to a packet coming into your old colo and being NATed to the completely different set of IPs at your new colo ... - packet from client arrives at oldcolo - packet is DNATed to newcolo - newcolo receives packet with source address client, destination address newcolo (self) - newcolo responds to packet by looking in it's routing table, it sees that it is responding to a non-local source address and so replies via it's default route Basically, the client would initiate communications with oldcolo but recieve replies with from the address of newcolo. The simple way around this is to use a proxy as other people have suggested. Personally I wouldn't bother with supporting things on the old address, set TTLs on the A records very low (let's say 10 minutes) at the point where you wish to switch the servers and just do it. If DNS is done correctly then there should be very little downtime. -- Fraser Campbell [EMAIL PROTECTED] http://www.wehave.net/ Georgetown, Ontario, Canada Debian GNU/Linux
Re: how to relocate servers transparently
On Monday 14 June 2004 14:57, Adrian 'Dagurashibanipal' von Bidder wrote: This may be obvious, but not to me... is there any difference compared to using iptables DNAT? At the time I was using rinetd/netcat to do this kind of work, I didn't know enough about iptables and the SNAT/DNAT pair which are required to do this kind of proxying. For lightweight traffic I often still choose this route, just out of habit :) Cheers, Gavin.
Re: how to relocate servers transparently
Hello Adrian! NAT is only working with IPs directly connected to your ethernet card. or NAT-ed IPs not arbitrary ones. Or? rgds, j. Adrian 'Dagurashibanipal' von Bidder wrote: On Monday 14 June 2004 14.59, Gavin Hamill wrote: Yep, rinetd is a simple userspace proxy. I used it for a while, but it had a few unpleasant 'features' I didn't like (we were using dynamic DNS with it, so I expect you wouldn't encounter them) Once our 'remote points' were static IPs, I switched to using 'netcat' instead. [...] This may be obvious, but not to me... is there any difference compared to using iptables DNAT? cheers -- vbi -- Andreas John net-lab GmbH Luisenstrasse 30b 63067 Offenbach Tel: +49 69 85700331 http://www.net-lab.net
