Maybe I didn't express myself clearly. When I said DNS RR, I meant its simplest implementation which resolves a hostname to multiple IPs.
Whatever method you use to map host names to IPs, the problem is that the zookeeper client code will always cache the IPs. So to be able to swap out a machine, all clients would have to be restarted, which if you have 100s of clients, is a major pain. If you want to move the entire cluster to new machines, this becomes even harder. I don't see why re-resolving host names to IPs in the reconnect logic is a problem for zookeeper, since you shuffle the list of IPs anyways. Thanks, Neha On Mon, Jan 9, 2012 at 10:31 AM, Camille Fournier <cami...@apache.org> wrote: > You can't sensibly round robin within the client code if you re-resolve on > every reconnect, if you're using dns rr. If that's your goal you'd want a > list of dns alias names and re-resolve each hostname when you hit it on > reconnect. But that will break people using dns rr. > You can look into writing a pluggable reconnect logic into the zk client, > that's what would be required to do this but at the end of the day you'll > have to give your users special clients to make that work. > > C > On Jan 9, 2012 1:16 PM, "Neha Narkhede" <neha.narkh...@gmail.com> wrote: > >> I was reading through the client code and saw that zookeeper client >> caches the server IPs during startup and maintains it for the rest of >> its lifetime. If we go with the DNS RR approach or a load balancer >> approach, and later swap out a server with a new one ( with a new IP >> ), all clients would have to be restarted to be able to "forget" the >> old IP and see the new one. That doesn't look like a clean approach to >> such upgrades. One way of getting around this problem, is adding the >> resolution of host names to IPs in the "reconnect" logic in addition >> to the constructor. So when such upgrades happen and the client >> reconnects, it will see the new list of IPs, and wouldn't require to >> be restarted. >> >> Does this approach sound good or am I missing something here ? >> >> Thanks, >> Neha >> >> On Wed, Dec 21, 2011 at 7:21 PM, Camille Fournier <cami...@apache.org> >> wrote: >> > DNS RR is good. I had good experiences using that for my client >> > configs for exactly the reasons you are listing. >> > >> > On Wed, Dec 21, 2011 at 8:43 PM, Neha Narkhede <neha.narkh...@gmail.com> >> wrote: >> >> Thanks for the responses! >> >> >> >>>> How are your clients configured to find the zks now? >> >> >> >> Our clients currently use the list of hostnames and ports that >> >> comprise the zookeeper cluster. For example, >> >> zoo1:port1,zoo2:port2,zoo3:port3 >> >> >> >>>> > - switch DNS, >> >>> - wait for caches to die, >> >> >> >> This is something we thought about however, if I understand it >> >> correctly, doesn't JVM cache DNS entries forever until it is restarted >> >> ? We haven't specifically turned DNS caching off on our clients. So >> >> this solution would require us to restart the clients to see the new >> >> list of zookeeper hosts. >> >> >> >> Another thought is to use DNS RR and have the client zk url have one >> >> name that resolves to and returns a list of IPs to the zookeeper >> >> client. This has the advantage of being able to perform hardware >> >> migration without changing the client connection url, in the future. >> >> Do people have thoughts about using a DNS RR ? >> >> >> >> Thanks, >> >> Neha >> >> >> >> On Tue, Dec 20, 2011 at 1:06 PM, Ted Dunning <ted.dunn...@gmail.com> >> wrote: >> >>> In particular, aren't you using DNS names? If you are, then you can >> >>> >> >>> - expand the quorum with the new hardware on new IP addresses, >> >>> - switch DNS, >> >>> - wait for caches to die, >> >>> - restart applications without reconfig or otherwise force new >> connections, >> >>> - decrease quorum size again >> >>> >> >>> On Tue, Dec 20, 2011 at 12:26 PM, Camille Fournier <cami...@apache.org >> >wrote: >> >>> >> >>>> How are your clients configured to find the zks now? How many clients >> do >> >>>> you have? >> >>>> >> >>>> From my phone >> >>>> On Dec 20, 2011 3:14 PM, "Neha Narkhede" <neha.narkh...@gmail.com> >> wrote: >> >>>> >> >>>> > Hi, >> >>>> > >> >>>> > As part of upgrading to Zookeeper 3.3.4, we also have to migrate our >> >>>> > zookeeper cluster to new hardware. I'm trying to figure out the best >> >>>> > strategy to achieve that with no downtime. >> >>>> > Here are some possible solutions I see at the moment, I could have >> >>>> > missed a few though - >> >>>> > >> >>>> > 1. Swap each machine out with a new machine, but with the same >> host/IP. >> >>>> > >> >>>> > Pros: No client side config needs to be changed. >> >>>> > Cons: Relatively tedious task for Operations >> >>>> > >> >>>> > 2. Add new machines, with different host/IPs to the existing >> cluster, >> >>>> > and remove the older machines, taking care to maintain the quorum at >> >>>> > all times >> >>>> > >> >>>> > Pros: Easier for Operations >> >>>> > Cons: Client side configs need to be changed and clients need to be >> >>>> > restarted/bounced. Another problem is having a large quorum for >> >>>> > sometime (potentially 9 nodes). >> >>>> > >> >>>> > 3. Hide the new cluster behind either a Hardware load balancer or a >> >>>> > DNS server resolving to all host ips. >> >>>> > >> >>>> > Pros: Makes it easier to move hardware around in the future >> >>>> > Cons: Possible timeout issues with load balancers messing with >> >>>> > zookeeper functionality or performance >> >>>> > >> >>>> > Read this and found it helpful - >> >>>> > >> >>>> > >> >>>> >> http://apache.markmail.org/message/44tbj53q2jufplru?q=load+balancer+list:org%2Eapache%2Ehadoop%2Ezookeeper-user&page=1 >> >>>> > But would like to hear from the authors and the users who might have >> >>>> > tried this in a real production setup. >> >>>> > >> >>>> > I'm very interested in finding a long term solution for masking the >> >>>> > zookeeper host names. Any inputs here are appreciated ! >> >>>> > >> >>>> > In addition to this, it will also be great to know what people think >> >>>> > about options 1 and 2, as a solution for hardware changes in >> >>>> > Zookeeper. >> >>>> > >> >>>> > Thanks, >> >>>> > Neha >> >>>> > >> >>>> >>
