How safe is "nodetool move" in 1.2 ?

[Oleg Dulin]

I need to rebalance my cluster. I am sure this question has been asked 
before -- will 1.2 continue to serve reads and writes correctly while 
move is in progress ?


Need this for my sanity.
--
Regards,
Oleg Dulin
http://www.olegdulin.com

Re: How safe is "nodetool move" in 1.2 ?

[Jonathan Lacefield]

Assuming you have enough nodes not undergoing "move" to meet your CL
requirements, then yes, your cluster will still accept reads and writes.
However, it's always good to test this before doing it in production to
ensure your cluster and app will function as designed.

Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487






On Wed, Apr 16, 2014 at 7:57 AM, Oleg Dulin  wrote:

> I need to rebalance my cluster. I am sure this question has been asked
> before -- will 1.2 continue to serve reads and writes correctly while move
> is in progress ?
>
> Need this for my sanity.
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>

Re: How safe is "nodetool move" in 1.2 ?

[Richard Low]

On 16 April 2014 05:08, Jonathan Lacefield  wrote:

> Assuming you have enough nodes not undergoing "move" to meet your CL
> requirements, then yes, your cluster will still accept reads and writes.
> However, it's always good to test this before doing it in production to
> ensure your cluster and app will function as designed.
>

This is not a correctness requirement: writes go to the move source and
destination during the move and reads come from the source. Otherwise you
could lose data during move (and certainly would lose data if replication
factor was one). However, nodes that are involved in the move will be
slower so it will be better for performance to not move nodes that share
replicas simultaneously.

Richard.

Re: How safe is "nodetool move" in 1.2 ?

[Colin]

I have recently tested this scenario under a couple versions of Cassandra and 
have been able to write and read to/from the cluster while performing a move.

I performed these tests utilizing an RF=2 on a three node cluster while 
performing quorum reads and received no errors due to unavailable replicas.

I will be doing some more testing on this under different scenarios, but so far 
so good

However, I would strongly recommend an RF of at least 3 when performing quorum 
based reads because otherwise you're subject to failed reads in event of losing 
one node.

--
Colin
320-221-9531


> On Apr 16, 2014, at 6:28 PM, Richard Low  wrote:
> 
>> On 16 April 2014 05:08, Jonathan Lacefield  wrote:
>> Assuming you have enough nodes not undergoing "move" to meet your CL 
>> requirements, then yes, your cluster will still accept reads and writes.   
>> However, it's always good to test this before doing it in production to 
>> ensure your cluster and app will function as designed.
> 
> This is not a correctness requirement: writes go to the move source and 
> destination during the move and reads come from the source. Otherwise you 
> could lose data during move (and certainly would lose data if replication 
> factor was one). However, nodes that are involved in the move will be slower 
> so it will be better for performance to not move nodes that share replicas 
> simultaneously.
> 
> Richard.

Re: How safe is "nodetool move" in 1.2 ?

[Robert Coli]

On Wed, Apr 16, 2014 at 4:57 AM, Oleg Dulin  wrote:

> I need to rebalance my cluster. I am sure this question has been asked
> before -- will 1.2 continue to serve reads and writes correctly while move
> is in progress ?
>

Yes, but "move" is subject to CASSANDRA-2434 until that bug is resolved. Of
course, so is bootstrap, so that's probably "ok." :)

https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13104715&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13104715

=Rob