Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-10 Thread Patrick Hunt
Looks like to me. I marked it as such. Patrick On Mon, Jan 9, 2012 at 6:49 PM, Neha Narkhede wrote: > Patrick, > > Looks like https://issues.apache.org/jira/browse/ZOOKEEPER-1356 is a > duplicate of 338 ? If yes, then I'll mark it to reflect the same. > > Thanks, > Neha > > On Mon, Jan 9, 2012 a

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Neha Narkhede
Patrick, Looks like https://issues.apache.org/jira/browse/ZOOKEEPER-1356 is a duplicate of 338 ? If yes, then I'll mark it to reflect the same. Thanks, Neha On Mon, Jan 9, 2012 at 5:36 PM, Patrick Hunt wrote: > dup of https://issues.apache.org/jira/browse/ZOOKEEPER-338 ? > > Patrick > > On Mon,

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Patrick Hunt
dup of https://issues.apache.org/jira/browse/ZOOKEEPER-338 ? Patrick On Mon, Jan 9, 2012 at 3:17 PM, Ted Dunning wrote: > Neha > > Filing a jira is a great way to further the discussion. > > Sent from my iPhone > > On Jan 9, 2012, at 15:33, Neha Narkhede wrote: > If you just have machine n

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Ted Dunning
Neha Filing a jira is a great way to further the discussion. Sent from my iPhone On Jan 9, 2012, at 15:33, Neha Narkhede wrote: >>> If you just have machine names in a list that you pass in, then yes, we > could re-resolve on every reconnect and you could just re-alias that name > to a new I

RE: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Alexander Shraer
om] On Behalf Of > Camille Fournier > Sent: Monday, January 09, 2012 12:47 PM > To: dev@zookeeper.apache.org > Subject: Re: Performing no downtime hardware changes to a live > zookeeper cluster > > Sounds fine with me, probably should make it a flaggable option. > > C

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Camille Fournier
Sounds fine with me, probably should make it a flaggable option. C On Mon, Jan 9, 2012 at 3:33 PM, Neha Narkhede wrote: > >> If you just have machine names in a list that you pass in, then yes, we > could re-resolve on every reconnect and you could just re-alias that name > to a new IP. But you

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Neha Narkhede
>> If you just have machine names in a list that you pass in, then yes, we could re-resolve on every reconnect and you could just re-alias that name to a new IP. But you'll have to put in logic that will do that but not break people using DNS RR. Having a list of machine names that can be changed

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Camille Fournier
We don't shuffle IPs after the initial resolution of IP addresses. In DNS RR, you resolve to a list of IPs, shuffle these, and then we round robin through them trying to connect. If you re-resolve on every round-robin, you have to put in logic to know which ones have changed and somehow maintain t

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Neha Narkhede
Maybe I didn't express myself clearly. When I said DNS RR, I meant its simplest implementation which resolves a hostname to multiple IPs. Whatever method you use to map host names to IPs, the problem is that the zookeeper client code will always cache the IPs. So to be able to swap out a machine,

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Camille Fournier
You can't sensibly round robin within the client code if you re-resolve on every reconnect, if you're using dns rr. If that's your goal you'd want a list of dns alias names and re-resolve each hostname when you hit it on reconnect. But that will break people using dns rr. You can look into writing

Re: Performing no downtime hardware changes to a live zookeeper cluster

2012-01-09 Thread Neha Narkhede
I was reading through the client code and saw that zookeeper client caches the server IPs during startup and maintains it for the rest of its lifetime. If we go with the DNS RR approach or a load balancer approach, and later swap out a server with a new one ( with a new IP ), all clients would have

Re: Performing no downtime hardware changes to a live zookeeper cluster

2011-12-21 Thread Camille Fournier
DNS RR is good. I had good experiences using that for my client configs for exactly the reasons you are listing. On Wed, Dec 21, 2011 at 8:43 PM, Neha Narkhede wrote: > Thanks for the responses! > >>> How are your clients configured to find the zks now? > > Our clients currently use the list of h

Re: Performing no downtime hardware changes to a live zookeeper cluster

2011-12-21 Thread Neha Narkhede
Thanks for the responses! >> How are your clients configured to find the zks now? Our clients currently use the list of hostnames and ports that comprise the zookeeper cluster. For example, zoo1:port1,zoo2:port2,zoo3:port3 >> > - switch DNS, > - wait for caches to die, This is something we thou

Re: Performing no downtime hardware changes to a live zookeeper cluster

2011-12-20 Thread Ted Dunning
In particular, aren't you using DNS names? If you are, then you can - expand the quorum with the new hardware on new IP addresses, - switch DNS, - wait for caches to die, - restart applications without reconfig or otherwise force new connections, - decrease quorum size again On Tue, Dec 20, 2011

Re: Performing no downtime hardware changes to a live zookeeper cluster

2011-12-20 Thread Camille Fournier
How are your clients configured to find the zks now? How many clients do you have? >From my phone On Dec 20, 2011 3:14 PM, "Neha Narkhede" wrote: > Hi, > > As part of upgrading to Zookeeper 3.3.4, we also have to migrate our > zookeeper cluster to new hardware. I'm trying to figure out the best

Performing no downtime hardware changes to a live zookeeper cluster

2011-12-20 Thread Neha Narkhede
Hi, As part of upgrading to Zookeeper 3.3.4, we also have to migrate our zookeeper cluster to new hardware. I'm trying to figure out the best strategy to achieve that with no downtime. Here are some possible solutions I see at the moment, I could have missed a few though - 1. Swap each machine ou