Re: [Gluster-users] Replace brick of a dead node
On Friday 12 August 2011 22:58:22 Rajat Chopra wrote: > Is there a way I can achieve that without any downtime? Maybe [1] could help? [1] http://bugs.gluster.com/show_bug.cgi?id=2506#c3 best regards, Marcel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replace brick of a dead node
On Fri, Aug 12, 2011 at 4:49 PM, Rajat Chopra wrote: > > I am afraid the 'replace-brick' procedure does not work well if the node is > dead. Here is the (long-ish) step-wise procedure for the dead-end that I run > into... > > [node-1 $] service glusterd start > [node-1 $] gluster volume create my-vol replica 2 node-1:/srv-node-1-first > node-1:/srv-node-1-second > [node-1 $] gluster volume start my-vol > # this began my gluster service on first node with two bricks replicated but > sourcing from the same node > # next I add a new node and replace one of the bricks with a new brick > location on second node > # the purpose is to achieve failover redundancy > [node-2 $] service glusterd start > [node-1 $] gluster peer probe node-2 > [node-2 $] gluster peer probe node-1 > [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-second > node-2:/srv-node-2-third start > # this starts the replace operation and after a while I can do volume info > from either node > [node-2 $] gluster volume info > Volume Name: my-vol > Type: Replicate > Status: Started > Number of Bricks: 2 > Transport-type: tcp > Bricks: > Brick1: node-1:/srv-node-1-first > Brick2: node-2:/srv-node-2-third > > # all good so far... now node-1 dies (no EBS, no disk, no data... just not > reachable.. its a pvt cloud and the machine running the vm had a hardware > failure) > # good gluster serves well from node-2 to all the clients nicely too > # now I want to replace the node-1 brick to another brick in node-2 so that I > can pass it on to new nodes later > > # so according to the suggestion, I ran replace-brick command > [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-first > node-2:/srv-node-2-fourth start Did you run $ gluster volume replace-brick my-vol node-1:/srv-node-1-first node-2:/srv-node-2-fourth commit ? If not, try running this additional command commit. This will make necessary changes to the config. So don't check status but run commit right after because dead node is not around. > # the command succeeds without errors, so I check status... > [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-first > node-2:/srv-node-2-fourth status > # this command is supposed to return the status, but it returns nothing > # I check with gluster volume info on node-2 > [node-2 $] gluster volume info > No volumes present > # wha?? where did my volume go? > # Note that all this while.. my mounted client is working fine, so no downtime > > Since 'gluster volume info' returned with 'No volumes present', I assume that > the procedure does not work. Is there something wrong in my procedure, or was > it not supposed to work anyway? > > I am using v3.1.1 > > Again, I really appreciate the help, but I seem to be stuck. The email > suggested the procedure in the following link > [ > http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server > ] > It seems like a better way of replacing dead-nodes, but then it would seem > that I cant replace the brick from dead node to a newly created path on an > existing node, because I should have the hostname matched. That is fine too, > if its a requirement, but can I assume that it will work with 3.1.1 or do I > have to upgrade to 3.2 for it? > > Thanks again for the assistance. > Rajat > > > > > > - Original Message - > From: "Mohit Anchlia" > To: "Rajat Chopra" > Cc: "Harshavardhana" , gluster-users@gluster.org > Sent: Friday, August 12, 2011 3:07:59 PM > Subject: Re: [Gluster-users] Replace brick of a dead node > > On Fri, Aug 12, 2011 at 2:35 PM, Rajat Chopra wrote: >> >> Thank you Harsha for the quick response. >> >> Unfortunately, the infrastructure is in the cloud. So, I cant get the dead >> node's disk. >> Since I have replication 'ON', there is no downtime as the brick on the >> second node serves well, but I want the redundancy/replication to be >> restored with the introduction of a new node (#3) in the cluster. > > One way is > http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server > > Other way is to use replace-brick. You should be able to use it even > if the node is dead. > >> >> I would hope there is a gluster command to just forget about the dead node's >> brick, and pick up the new brick and start replicating/serving from the new >> location (in conjunction with the one existing brick on the #2 node). Is >> that the self heal feature? I am using v3.11 as of now. >> &g
Re: [Gluster-users] Replace brick of a dead node
I am afraid the 'replace-brick' procedure does not work well if the node is dead. Here is the (long-ish) step-wise procedure for the dead-end that I run into... [node-1 $] service glusterd start [node-1 $] gluster volume create my-vol replica 2 node-1:/srv-node-1-first node-1:/srv-node-1-second [node-1 $] gluster volume start my-vol # this began my gluster service on first node with two bricks replicated but sourcing from the same node # next I add a new node and replace one of the bricks with a new brick location on second node # the purpose is to achieve failover redundancy [node-2 $] service glusterd start [node-1 $] gluster peer probe node-2 [node-2 $] gluster peer probe node-1 [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-second node-2:/srv-node-2-third start # this starts the replace operation and after a while I can do volume info from either node [node-2 $] gluster volume info Volume Name: my-vol Type: Replicate Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: node-1:/srv-node-1-first Brick2: node-2:/srv-node-2-third # all good so far... now node-1 dies (no EBS, no disk, no data... just not reachable.. its a pvt cloud and the machine running the vm had a hardware failure) # good gluster serves well from node-2 to all the clients nicely too # now I want to replace the node-1 brick to another brick in node-2 so that I can pass it on to new nodes later # so according to the suggestion, I ran replace-brick command [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-first node-2:/srv-node-2-fourth start # the command succeeds without errors, so I check status... [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-first node-2:/srv-node-2-fourth status # this command is supposed to return the status, but it returns nothing # I check with gluster volume info on node-2 [node-2 $] gluster volume info No volumes present # wha?? where did my volume go? # Note that all this while.. my mounted client is working fine, so no downtime Since 'gluster volume info' returned with 'No volumes present', I assume that the procedure does not work. Is there something wrong in my procedure, or was it not supposed to work anyway? I am using v3.1.1 Again, I really appreciate the help, but I seem to be stuck. The email suggested the procedure in the following link [ http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server ] It seems like a better way of replacing dead-nodes, but then it would seem that I cant replace the brick from dead node to a newly created path on an existing node, because I should have the hostname matched. That is fine too, if its a requirement, but can I assume that it will work with 3.1.1 or do I have to upgrade to 3.2 for it? Thanks again for the assistance. Rajat - Original Message - From: "Mohit Anchlia" To: "Rajat Chopra" Cc: "Harshavardhana" , gluster-users@gluster.org Sent: Friday, August 12, 2011 3:07:59 PM Subject: Re: [Gluster-users] Replace brick of a dead node On Fri, Aug 12, 2011 at 2:35 PM, Rajat Chopra wrote: > > Thank you Harsha for the quick response. > > Unfortunately, the infrastructure is in the cloud. So, I cant get the dead > node's disk. > Since I have replication 'ON', there is no downtime as the brick on the > second node serves well, but I want the redundancy/replication to be restored > with the introduction of a new node (#3) in the cluster. One way is http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server Other way is to use replace-brick. You should be able to use it even if the node is dead. > > I would hope there is a gluster command to just forget about the dead node's > brick, and pick up the new brick and start replicating/serving from the new > location (in conjunction with the one existing brick on the #2 node). Is that > the self heal feature? I am using v3.11 as of now. > > Rajat > > > > > - Original Message - > From: "Harshavardhana" > To: "Rajat Chopra" > Cc: gluster-users@gluster.org > Sent: Friday, August 12, 2011 2:06:14 PM > Subject: Re: [Gluster-users] Replace brick of a dead node > >> I have a two node cluster, with two bricks replicated, one on each node. >> Lets say one of the node dies and is unreachable. > > If you have the disk from the dead node, then all have to do is plug > it in new system and start running following commands. > > gluster volume replace-brickstart > gluster volume replace-brickcommit > > You don't have to migrate the data, this works as expected. > > Since you have a replicate you wouldn't see a downtime, but mind you > self-heal will
Re: [Gluster-users] Replace brick of a dead node
On Fri, Aug 12, 2011 at 4:03 PM, Harshavardhana wrote: >> Since I have replication 'ON', there is no downtime as the brick on the >> second node serves well, but I want the redundancy/replication to be >> restored with the introduction of a new node (#3) in the cluster. >> > > Exactly if its in Cloud then the disk be it EBS blocks which can be > reattached back to the new server and you can do a replace-brick even > when the old-brick is dead/unreachable. Here old-brick i meant 'old instance' not 'disk' per se. -Harsha ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replace brick of a dead node
> Since I have replication 'ON', there is no downtime as the brick on the > second node serves well, but I want the redundancy/replication to be restored > with the introduction of a new node (#3) in the cluster. > Exactly if its in Cloud then the disk be it EBS blocks which can be reattached back to the new server and you can do a replace-brick even when the old-brick is dead/unreachable. If there are no EBS blocks, then sure there should be a mechanism to reattach the brick associated with that instance. > I would hope there is a gluster command to just forget about the dead node's > brick, and pick up the new brick and start replicating/serving from the new Gluster cannot decide as it has no awareness that it is in Cloud or a Bare metal or a KVM setup. So right now the above procedure stands good. -Harsha ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replace brick of a dead node
On Fri, Aug 12, 2011 at 2:35 PM, Rajat Chopra wrote: > > Thank you Harsha for the quick response. > > Unfortunately, the infrastructure is in the cloud. So, I cant get the dead > node's disk. > Since I have replication 'ON', there is no downtime as the brick on the > second node serves well, but I want the redundancy/replication to be restored > with the introduction of a new node (#3) in the cluster. One way is http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server Other way is to use replace-brick. You should be able to use it even if the node is dead. > > I would hope there is a gluster command to just forget about the dead node's > brick, and pick up the new brick and start replicating/serving from the new > location (in conjunction with the one existing brick on the #2 node). Is that > the self heal feature? I am using v3.11 as of now. > > Rajat > > > > > - Original Message - > From: "Harshavardhana" > To: "Rajat Chopra" > Cc: gluster-users@gluster.org > Sent: Friday, August 12, 2011 2:06:14 PM > Subject: Re: [Gluster-users] Replace brick of a dead node > >> I have a two node cluster, with two bricks replicated, one on each node. >> Lets say one of the node dies and is unreachable. > > If you have the disk from the dead node, then all have to do is plug > it in new system and start running following commands. > > gluster volume replace-brickstart > gluster volume replace-brickcommit > > You don't have to migrate the data, this works as expected. > > Since you have a replicate you wouldn't see a downtime, but mind you > self-heal will kick in as of 3.2 it will be blocking, wait for 3.3 you > have non-blocking self-healing capabilities. > >> I want to be able to spin a new node and replace the dead node's brick to a >> location on the new node. > > This is out of Gluster's hand, if you already have mechanisms to > decommission a brick and reattach it on new node then above steps are > fairly simple. > > Go ahead and try it, it should work. > > -Harsha > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replace brick of a dead node
Thank you Harsha for the quick response. Unfortunately, the infrastructure is in the cloud. So, I cant get the dead node's disk. Since I have replication 'ON', there is no downtime as the brick on the second node serves well, but I want the redundancy/replication to be restored with the introduction of a new node (#3) in the cluster. I would hope there is a gluster command to just forget about the dead node's brick, and pick up the new brick and start replicating/serving from the new location (in conjunction with the one existing brick on the #2 node). Is that the self heal feature? I am using v3.11 as of now. Rajat - Original Message - From: "Harshavardhana" To: "Rajat Chopra" Cc: gluster-users@gluster.org Sent: Friday, August 12, 2011 2:06:14 PM Subject: Re: [Gluster-users] Replace brick of a dead node > I have a two node cluster, with two bricks replicated, one on each node. > Lets say one of the node dies and is unreachable. If you have the disk from the dead node, then all have to do is plug it in new system and start running following commands. gluster volume replace-brickstart gluster volume replace-brickcommit You don't have to migrate the data, this works as expected. Since you have a replicate you wouldn't see a downtime, but mind you self-heal will kick in as of 3.2 it will be blocking, wait for 3.3 you have non-blocking self-healing capabilities. > I want to be able to spin a new node and replace the dead node's brick to a > location on the new node. This is out of Gluster's hand, if you already have mechanisms to decommission a brick and reattach it on new node then above steps are fairly simple. Go ahead and try it, it should work. -Harsha ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replace brick of a dead node
> I have a two node cluster, with two bricks replicated, one on each node. > Lets say one of the node dies and is unreachable. If you have the disk from the dead node, then all have to do is plug it in new system and start running following commands. gluster volume replace-brickstart gluster volume replace-brickcommit You don't have to migrate the data, this works as expected. Since you have a replicate you wouldn't see a downtime, but mind you self-heal will kick in as of 3.2 it will be blocking, wait for 3.3 you have non-blocking self-healing capabilities. > I want to be able to spin a new node and replace the dead node's brick to a > location on the new node. This is out of Gluster's hand, if you already have mechanisms to decommission a brick and reattach it on new node then above steps are fairly simple. Go ahead and try it, it should work. -Harsha ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Replace brick of a dead node
Hi! Seeking pardon from the experts, but I have a basic usage question that I could not find a straightforward answer to. I have a two node cluster, with two bricks replicated, one on each node. Lets say one of the node dies and is unreachable. I want to be able to spin a new node and replace the dead node's brick to a location on the new node. The command 'gluster volume replace-brick' wont work because it cant reach the source brick. Is there a way I can achieve that without any downtime? Thanks for the help. Rajat ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
