[jira] [Comment Edited] (CASSANDRA-10887) Pending range calculator gives wrong pending ranges for moves
[ https://issues.apache.org/jira/browse/CASSANDRA-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086543#comment-15086543 ] sankalp kohli edited comment on CASSANDRA-10887 at 1/7/16 12:09 AM: I have uploaded patches for 2.2 and 3.0. This is a major bug and should go in every branch/release possible. Let me know if you need a patch for any other branch was (Author: kohlisankalp): I have uploaded patches for 2.2 and 3.0. This is a major bug and should go in every branch/release possible. > Pending range calculator gives wrong pending ranges for moves > - > > Key: CASSANDRA-10887 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10887 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Richard Low >Assignee: sankalp kohli >Priority: Critical > Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x > > Attachments: CASSANDRA-10887.diff, CASSANDRA_10887_2.2.diff, > CASSANDRA_10887_3.0.diff, CASSANDRA_10887_v2.diff, CASSANDRA_10887_v3.diff > > > My understanding is the PendingRangeCalculator is meant to calculate who > should receive extra writes during range movements. However, it adds the > wrong ranges for moves. An extreme example of this can be seen in the > following reproduction. Create a 5 node cluster (I did this on 2.0.16 and > 2.2.4) and a keyspace RF=3 and a simple table. Then start moving a node and > immediately kill -9 it. Now you see a node as down and moving in the ring. > Try a quorum write for a partition that is stored on that node - it will fail > with a timeout. Further, all CAS reads or writes fail immediately with > unavailable exception because they attempt to include the moving node twice. > This is likely to be the cause of CASSANDRA-10423. > In my example I had this ring: > 127.0.0.1 rack1 Up Normal 170.97 KB 20.00% > -9223372036854775808 > 127.0.0.2 rack1 Up Normal 124.06 KB 20.00% > -5534023222112865485 > 127.0.0.3 rack1 Down Moving 108.7 KB40.00% > 1844674407370955160 > 127.0.0.4 rack1 Up Normal 142.58 KB 0.00% > 1844674407370955161 > 127.0.0.5 rack1 Up Normal 118.64 KB 20.00% > 5534023222112865484 > Node 3 was moving to -1844674407370955160. I added logging to print the > pending and natural endpoints. For ranges owned by node 3, node 3 appeared in > pending and natural endpoints. The blockFor is increased to 3 so we’re > effectively doing CL.ALL operations. This manifests as write timeouts and CAS > unavailables when the node is down. > The correct pending range for this scenario is node 1 is gaining the range > (-1844674407370955160, 1844674407370955160). So node 1 should be added as a > destination for writes and CAS for this range, not node 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10887) Pending range calculator gives wrong pending ranges for moves
[ https://issues.apache.org/jira/browse/CASSANDRA-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064508#comment-15064508 ] Philip Thompson edited comment on CASSANDRA-10887 at 12/18/15 9:44 PM: --- I've written up a dtest here: https://github.com/riptano/cassandra-dtest/pull/717 and pinged Branimir about it. I can't reproduce manually with Simon's steps or with my test on 2.1, 2.2, or 3.0. I've asked [~mambocab] to take a look as well. I did also run against 2.0.16, to be clear. was (Author: philipthompson): I've written up a dtest here: https://github.com/riptano/cassandra-dtest/pull/717 and pinged Branimir about it. I can't reproduce manually with Simon's steps or with my test on 2.1, 2.2, or 3.0. I've asked [~mambocab] to take a look as well. > Pending range calculator gives wrong pending ranges for moves > - > > Key: CASSANDRA-10887 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10887 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Richard Low >Assignee: Branimir Lambov >Priority: Critical > > My understanding is the PendingRangeCalculator is meant to calculate who > should receive extra writes during range movements. However, it adds the > wrong ranges for moves. An extreme example of this can be seen in the > following reproduction. Create a 5 node cluster (I did this on 2.0.16 and > 2.2.4) and a keyspace RF=3 and a simple table. Then start moving a node and > immediately kill -9 it. Now you see a node as down and moving in the ring. > Try a quorum write for a partition that is stored on that node - it will fail > with a timeout. Further, all CAS reads or writes fail immediately with > unavailable exception because they attempt to include the moving node twice. > This is likely to be the cause of CASSANDRA-10423. > In my example I had this ring: > 127.0.0.1 rack1 Up Normal 170.97 KB 20.00% > -9223372036854775808 > 127.0.0.2 rack1 Up Normal 124.06 KB 20.00% > -5534023222112865485 > 127.0.0.3 rack1 Down Moving 108.7 KB40.00% > 1844674407370955160 > 127.0.0.4 rack1 Up Normal 142.58 KB 0.00% > 1844674407370955161 > 127.0.0.5 rack1 Up Normal 118.64 KB 20.00% > 5534023222112865484 > Node 3 was moving to -1844674407370955160. I added logging to print the > pending and natural endpoints. For ranges owned by node 3, node 3 appeared in > pending and natural endpoints. The blockFor is increased to 3 so we’re > effectively doing CL.ALL operations. This manifests as write timeouts and CAS > unavailables when the node is down. > The correct pending range for this scenario is node 1 is gaining the range > (-1844674407370955160, 1844674407370955160). So node 1 should be added as a > destination for writes and CAS for this range, not node 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10887) Pending range calculator gives wrong pending ranges for moves
[ https://issues.apache.org/jira/browse/CASSANDRA-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064508#comment-15064508 ] Philip Thompson edited comment on CASSANDRA-10887 at 12/18/15 7:07 PM: --- I've written up a dtest here: https://github.com/riptano/cassandra-dtest/pull/717 and pinged Branimir about it. I can't reproduce manually with Simon's steps or with my test on 2.1, 2.2, or 3.0. I've asked [~mambocab] to take a look as well. was (Author: philipthompson): I've written up a dtest here: https://github.com/riptano/cassandra-dtest/pull/717 and pinged Branimir about it. I can't reproduce manually with Simon's steps or with my test on 2.1, 2.2, or 3.0. I've asked Jim to take a look as well. > Pending range calculator gives wrong pending ranges for moves > - > > Key: CASSANDRA-10887 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10887 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Richard Low >Assignee: Branimir Lambov >Priority: Critical > > My understanding is the PendingRangeCalculator is meant to calculate who > should receive extra writes during range movements. However, it adds the > wrong ranges for moves. An extreme example of this can be seen in the > following reproduction. Create a 5 node cluster (I did this on 2.0.16 and > 2.2.4) and a keyspace RF=3 and a simple table. Then start moving a node and > immediately kill -9 it. Now you see a node as down and moving in the ring. > Try a quorum write for a partition that is stored on that node - it will fail > with a timeout. Further, all CAS reads or writes fail immediately with > unavailable exception because they attempt to include the moving node twice. > This is likely to be the cause of CASSANDRA-10423. > In my example I had this ring: > 127.0.0.1 rack1 Up Normal 170.97 KB 20.00% > -9223372036854775808 > 127.0.0.2 rack1 Up Normal 124.06 KB 20.00% > -5534023222112865485 > 127.0.0.3 rack1 Down Moving 108.7 KB40.00% > 1844674407370955160 > 127.0.0.4 rack1 Up Normal 142.58 KB 0.00% > 1844674407370955161 > 127.0.0.5 rack1 Up Normal 118.64 KB 20.00% > 5534023222112865484 > Node 3 was moving to -1844674407370955160. I added logging to print the > pending and natural endpoints. For ranges owned by node 3, node 3 appeared in > pending and natural endpoints. The blockFor is increased to 3 so we’re > effectively doing CL.ALL operations. This manifests as write timeouts and CAS > unavailables when the node is down. > The correct pending range for this scenario is node 1 is gaining the range > (-1844674407370955160, 1844674407370955160). So node 1 should be added as a > destination for writes and CAS for this range, not node 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
