[ https://issues.apache.org/jira/browse/CASSANDRA-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085341#comment-14085341 ]
Michael Theroux commented on CASSANDRA-5264: -------------------------------------------- We have not upgraded to 1.2.X yet, so I won't be able to tell if its reproducible until we either upgrade, or setup a test environment. I'm not sure when we will be able to invest some time on it, but hopefully I could provide something in the next month or so. > Nodetool ring is reporting incorrect Effective ownership after upgrading from > 1.1.2 -> 1.1.9 > -------------------------------------------------------------------------------------------- > > Key: CASSANDRA-5264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5264 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.1.9 > Environment: Amazon AWS Linux, Large instance (8gig of RAM, ephemeral > storage). 12 Node cluster. Replication Factor 3, all queries performed with > LOCAL_QUORUM > Reporter: Michael Theroux > Priority: Minor > > We upgraded from Cassandra 1.1.2 to 1.1.9 yesterday. All indications are the > upgrade went well. Repair works as expected, and all our data is available. > Performance is as good, if not better, than it was previously. > However, nodetool ring is reporting inconsistent and incorrect results. This > was my ring information before the upgrade: > Address DC Rack Status State Load > Effective-Ownership Token > > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.22 us-east 1a Up Normal 77.75 GB 25.00% > Token(bytes[00000000000000000000000000000001]) > 10.0.10.23 us-east 1d Up Normal 82.68 GB 25.00% > Token(bytes[15555555555555555555555555555555]) > 10.0.8.20 us-east 1c Up Normal 81.72 GB 25.00% > Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa]) > 10.0.4.23 us-east 1a Up Normal 82.65 GB 25.00% > Token(bytes[40000000000000000000000000000000]) > 10.0.10.20 us-east 1d Up Normal 80.2 GB 25.00% > Token(bytes[55555555555555555555555555555554]) > 10.0.8.23 us-east 1c Up Normal 77.06 GB 25.00% > Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac]) > 10.0.4.21 us-east 1a Up Normal 81.37 GB 25.00% > Token(bytes[80000000000000000000000000000000]) > 10.0.10.24 us-east 1d Up Normal 83.37 GB 25.00% > Token(bytes[95555555555555555555555555555558]) > 10.0.8.21 us-east 1c Up Normal 84.33 GB 25.00% > Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.25 us-east 1a Up Normal 79.91 GB 25.00% > Token(bytes[c0000000000000000000000000000000]) > 10.0.10.21 us-east 1d Up Normal 83.46 GB 25.00% > Token(bytes[d5555555555555555555555555555558]) > 10.0.8.24 us-east 1c Up Normal 90.66 GB 25.00% > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > This is my ring information after the upgrade: > 10.0.4.22 us-east 1a Up Normal 77.74 GB 99.89% > Token(bytes[00000000000000000000000000000001]) > 10.0.10.23 us-east 1d Up Normal 82.82 GB 64.14% > Token(bytes[15555555555555555555555555555555]) > 10.0.8.20 us-east 1c Up Normal 81.89 GB 30.55% > Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa]) > 10.0.4.23 us-east 1a Up Normal 82.77 GB 0.04% > Token(bytes[40000000000000000000000000000000]) > 10.0.10.20 us-east 1d Up Normal 80.32 GB 0.04% > Token(bytes[55555555555555555555555555555554]) > 10.0.8.23 us-east 1c Up Normal 77.07 GB 0.04% > Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac]) > 10.0.4.21 us-east 1a Up Normal 81.35 GB 0.04% > Token(bytes[80000000000000000000000000000000]) > 10.0.10.24 us-east 1d Up Normal 83.49 GB 0.04% > Token(bytes[95555555555555555555555555555558]) > 10.0.8.21 us-east 1c Up Normal 84.47 GB 0.04% > Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.25 us-east 1a Up Normal 80.11 GB 0.04% > Token(bytes[c0000000000000000000000000000000]) > 10.0.10.21 us-east 1d Up Normal 83.5 GB 35.79% > Token(bytes[d5555555555555555555555555555558]) > 10.0.8.24 us-east 1c Up Normal 90.72 GB 69.38% > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > We use ByteOrderedPartitioning (we hash our own keys), and as you can see > from above, were achieving a somewhat equal distribution of data amongst the > nodes. > The node that always seems to own 99% of the keys is the node that I run > "nodetool ring" on. Running "nodetool ring" on two of these nodes at the > same time resulted in: > From 10.0.4.22: > Address DC Rack Status State Load > Effective-Ownership Token > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.22 us-east 1a Up Normal 77.72 GB 99.89% > Token(bytes[00000000000000000000000000000001]) > 10.0.10.23 us-east 1d Up Normal 82.74 GB 64.13% > Token(bytes[15555555555555555555555555555555]) > 10.0.8.20 us-east 1c Up Normal 81.79 GB 30.55% > Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa]) > 10.0.4.23 us-east 1a Up Normal 82.66 GB 0.04% > Token(bytes[40000000000000000000000000000000]) > 10.0.10.20 us-east 1d Up Normal 80.21 GB 0.04% > Token(bytes[55555555555555555555555555555554]) > 10.0.8.23 us-east 1c Up Normal 77.07 GB 0.04% > Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac]) > 10.0.4.21 us-east 1a Up Normal 81.38 GB 0.04% > Token(bytes[80000000000000000000000000000000]) > 10.0.10.24 us-east 1d Up Normal 83.43 GB 0.04% > Token(bytes[95555555555555555555555555555558]) > 10.0.8.21 us-east 1c Up Normal 84.42 GB 0.04% > Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.25 us-east 1a Up Normal 80.06 GB 0.04% > Token(bytes[c0000000000000000000000000000000]) > 10.0.10.21 us-east 1d Up Normal 83.49 GB 35.80% > Token(bytes[d5555555555555555555555555555558]) > 10.0.8.24 us-east 1c Up Normal 90.72 GB 69.37% > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > From 10.0.8.23: > Address DC Rack Status State Load > Effective-Ownership Token > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.22 us-east 1a Up Normal 77.72 GB 0.04% > Token(bytes[00000000000000000000000000000001]) > 10.0.10.23 us-east 1d Up Normal 82.78 GB 0.04% > Token(bytes[15555555555555555555555555555555]) > 10.0.8.20 us-east 1c Up Normal 81.79 GB 0.04% > Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa]) > 10.0.4.23 us-east 1a Up Normal 82.66 GB 33.84% > Token(bytes[40000000000000000000000000000000]) > 10.0.10.20 us-east 1d Up Normal 80.21 GB 67.51% > Token(bytes[55555555555555555555555555555554]) > 10.0.8.23 us-east 1c Up Normal 77.07 GB 99.89% > Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac]) > 10.0.4.21 us-east 1a Up Normal 81.38 GB 66.09% > Token(bytes[80000000000000000000000000000000]) > 10.0.10.24 us-east 1d Up Normal 83.43 GB 32.41% > Token(bytes[95555555555555555555555555555558]) > 10.0.8.21 us-east 1c Up Normal 84.42 GB 0.04% > Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > 10.0.4.25 us-east 1a Up Normal 80.06 GB 0.04% > Token(bytes[c0000000000000000000000000000000]) > 10.0.10.21 us-east 1d Up Normal 83.49 GB 0.04% > Token(bytes[d5555555555555555555555555555558]) > 10.0.8.24 us-east 1c Up Normal 90.72 GB 0.04% > Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8]) > I did a little digging, and do see some changes to how cassandra calculates > the splits in 1.1.8 as part of CASSANDRA-4803 (StorageService.getSplits()), > although I'm not familiar enough to tell if its a cause. -- This message was sent by Atlassian JIRA (v6.2#6252)