Re: Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-20 Thread Anand Somani
Thanks for the help, this seems to have worked. Except that while adding the new node we added the same token to a different IP (operational script goofup) and brought the node up, so now the other nodes just had the message that a new IP had taken over the token. - So we brought it down and f

Re: Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread Anand Somani
ok I will go with the IP change strategy and keep you posted. Not going to manually copy any data, just bring up the node and let it bootstrap. Thanks On Fri, Aug 19, 2011 at 11:46 AM, Peter Schuller < peter.schul...@infidyne.com> wrote: > > (Yes, this should definitely be easier. Maybe the most

Re: Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread Peter Schuller
> (Yes, this should definitely be easier. Maybe the most generally > useful fix would be for Cassandra to support a node joining the wring > in "write-only" mode. This would be useful in other cases, such as > when you're trying to temporarily off-load a node by dissabling > gossip). I knew I had

Re: Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread Peter Schuller
> From what I understand, Peter's recommendation should work for you. They > have both worked for me. No need to copy anything by hand on the new node. > Bootstrap/repair does that for you. From the Wiki: Right - it's just that the complication comes from the fact that he's using the same machine,

Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread Peter Schuller
> I am running read/write at quorum. At this point I have turned off my > clients from talking to this node. So if that is the case I can potentially > just nodetool repair (without changing IP). But would it be better if I No, other nodes in the cluster will still be sending reads to the node. >

Re: Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread jonathan . colby
Hi - From what I understand, Peter's recommendation should work for you. They have both worked for me. No need to copy anything by hand on the new node. Bootstrap/repair does that for you. From the Wiki: If a node goes down entirely, then you have two options: (Recommended approach) Bring

Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread Anand Somani
Let me be specific on lost data -> lost a replica , the other 2 nodes have replicas I am running read/write at quorum. At this point I have turned off my clients from talking to this node. So if that is the case I can potentially just nodetool repair (without changing IP). But would it be better

Re: Urgent:!! Re: Need to maintenance on a cassandra node, are there problems with this process

2011-08-19 Thread Peter Schuller
> ok, so we just lost the data on that node. are building the raid on it, but > once it is up what is the best way to bring it back in the cluster You're saying the raid failed and data is gone? > just let it come up and run nodetool repair > copy data from another node and then run nodetool repa