[ https://issues.apache.org/jira/browse/CASSANDRA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620628#comment-13620628 ]
Arya Goudarzi commented on CASSANDRA-5332: ------------------------------------------ Reopening as upgrading to 1.1.10 did not solve the problem. > 1.2.3 Nodetool reports 1.1.10 nodes as down > ------------------------------------------- > > Key: CASSANDRA-5332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5332 > Project: Cassandra > Issue Type: Bug > Environment: Sun Java 6_u39 > Ubuntu 10.04.1 > Cassandra 1.2.2 and 1.1.10 > Reporter: Arya Goudarzi > > During exercise of a rolling upgrade from 1.1.6 to 1.2.2, I upgraded one node > from 1.1.6 to 1.2.2 today. It appears that node running 1.2.2 reports nodes > in 1.1.6 as Down in nodetool. This means that the nodes running 1.1.6 see all > other nodes including 1.2.2 as Up. Here is the ring and gossip from nodes > with 1.1.6 for example. "231.121" is the upgraded node: > Address DC Rack Status State Load > Effective-Ownership Token > > 141784319550391026443072753098378663700 > XX.180.36 us-east 1b Up Normal 49.47 GB 25.00% > 1808575600 > XX.231.121 us-east 1c Up Normal 47.08 GB 25.00% > 7089215977519551322153637656637080005 > XX.177.177 us-east 1d Up Normal 33.64 GB 25.00% > 14178431955039102644307275311465584410 > XX.7.148 us-east 1b Up Normal 41.27 GB 25.00% > 42535295865117307932921825930779602030 > XX.20.9 us-east 1c Up Normal 38.51 GB 25.00% > 49624511842636859255075463585608106435 > XX.86.255 us-east 1d Up Normal 34.78 GB 25.00% > 56713727820156410577229101240436610840 > XX.63.230 us-east 1b Up Normal 38.11 GB 25.00% > 85070591730234615865843651859750628460 > XX.163.36 us-east 1c Up Normal 44.25 GB 25.00% > 92159807707754167187997289514579132865 > XX.31.234 us-east 1d Up Normal 44.66 GB 25.00% > 99249023685273718510150927169407637270 > XX.132.169 us-east 1b Up Normal 44.2 GB 25.00% > 127605887595351923798765477788721654890 > XX.71.63 us-east 1c Up Normal 38.74 GB 25.00% > 134695103572871475120919115443550159295 > XX.197.209 us-east 1d Up Normal 41.5 GB 25.00% > 141784319550391026443072753098378663700 > /XX.71.63 > RACK:1c > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.1598705272E10 > DC:us-east > INTERNAL_IP:XX.194.92 > STATUS:NORMAL,134695103572871475120919115443550159295 > RPC_ADDRESS:XX.194.92 > RELEASE_VERSION:1.1.6 > /XX.86.255 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:3.734334162E10 > DC:us-east > INTERNAL_IP:XX.6.195 > STATUS:NORMAL,56713727820156410577229101240436610840 > RPC_ADDRESS:XX.6.195 > RELEASE_VERSION:1.1.6 > /XX.7.148 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.4316975808E10 > DC:us-east > INTERNAL_IP:XX.47.250 > STATUS:NORMAL,42535295865117307932921825930779602030 > RPC_ADDRESS:XX.47.250 > RELEASE_VERSION:1.1.6 > /XX.63.230 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.0918593305E10 > DC:us-east > INTERNAL_IP:XX.89.127 > STATUS:NORMAL,85070591730234615865843651859750628460 > RPC_ADDRESS:XX.89.127 > RELEASE_VERSION:1.1.6 > /XX.132.169 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.745883458E10 > DC:us-east > INTERNAL_IP:XX.94.161 > STATUS:NORMAL,127605887595351923798765477788721654890 > RPC_ADDRESS:XX.94.161 > RELEASE_VERSION:1.1.6 > /XX.180.36 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:5.311963027E10 > DC:us-east > INTERNAL_IP:XX.123.112 > STATUS:NORMAL,1808575600 > RPC_ADDRESS:XX.123.112 > RELEASE_VERSION:1.1.6 > /XX.163.36 > RACK:1c > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.7516755022E10 > DC:us-east > INTERNAL_IP:XX.163.180 > STATUS:NORMAL,92159807707754167187997289514579132865 > RPC_ADDRESS:XX.163.180 > RELEASE_VERSION:1.1.6 > /XX.31.234 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.7954372912E10 > DC:us-east > INTERNAL_IP:XX.192.159 > STATUS:NORMAL,99249023685273718510150927169407637270 > RPC_ADDRESS:XX.192.159 > RELEASE_VERSION:1.1.6 > /XX.197.209 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.4558968005E10 > DC:us-east > INTERNAL_IP:XX.66.205 > STATUS:NORMAL,141784319550391026443072753098378663700 > RPC_ADDRESS:XX.66.205 > RELEASE_VERSION:1.1.6 > /XX.177.177 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:3.6115572697E10 > DC:us-east > INTERNAL_IP:XX.65.57 > STATUS:NORMAL,14178431955039102644307275311465584410 > RPC_ADDRESS:XX.65.57 > RELEASE_VERSION:1.1.6 > /XX.20.9 > RACK:1c > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.1352503882E10 > DC:us-east > INTERNAL_IP:XX.33.229 > STATUS:NORMAL,49624511842636859255075463585608106435 > RPC_ADDRESS:XX.33.229 > RELEASE_VERSION:1.1.6 > /XX.231.121 > RACK:1c > SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0 > X4:9c765678-d058-4d85-a588-638ce10ff984 > X3:7 > DC:us-east > INTERNAL_IP:XX.223.241 > RPC_ADDRESS:XX.223.241 > RELEASE_VERSION:1.2.2 > Now the nodetool on the 1.2.2 node shows all nodes as Down but itself. > Gossipinfo looks good though: > Datacenter: us-east > ========== > Replicas: 3 > Address Rack Status State Load Owns > Token > > 56713727820156410577229101240436610840 > XX.132.169 1b Down Normal 44.2 GB 25.00% > 127605887595351923798765477788721654890 > XX.7.148 1b Down Normal 41.27 GB 25.00% > 42535295865117307932921825930779602030 > XX.180.36 1b Down Normal 49.47 GB 25.00% > 1808575600 > XX.63.230 1b Down Normal 38.11 GB 25.00% > 85070591730234615865843651859750628460 > XX.231.121 1c Up Normal 47.25 GB 25.00% > 7089215977519551322153637656637080005 > XX.71.63 1c Down Normal 38.74 GB 25.00% > 134695103572871475120919115443550159295 > XX.177.177 1d Down Normal 33.64 GB 25.00% > 14178431955039102644307275311465584410 > XX.31.234 1d Down Normal 44.66 GB 25.00% > 99249023685273718510150927169407637270 > XX.20.9 1c Down Normal 38.51 GB 25.00% > 49624511842636859255075463585608106435 > XX.163.36 1c Down Normal 44.25 GB 25.00% > 92159807707754167187997289514579132865 > XX.197.209 1d Down Normal 41.5 GB 25.00% > 141784319550391026443072753098378663700 > XX.86.255 1d Down Normal 34.78 GB 25.00% > 56713727820156410577229101240436610840 > /XX.71.63 > RACK:1c > RPC_ADDRESS:XX.194.92 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.194.92 > STATUS:NORMAL,134695103572871475120919115443550159295 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.1598705272E10 > /XX.86.255 > RACK:1d > RPC_ADDRESS:XX.6.195 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.6.195 > STATUS:NORMAL,56713727820156410577229101240436610840 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:3.7343205002E10 > /XX.7.148 > RACK:1b > RPC_ADDRESS:XX.47.250 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.47.250 > STATUS:NORMAL,42535295865117307932921825930779602030 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.4316975808E10 > /XX.63.230 > RACK:1b > RPC_ADDRESS:XX.89.127 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.89.127 > STATUS:NORMAL,85070591730234615865843651859750628460 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.0918456687E10 > /XX.132.169 > RACK:1b > RPC_ADDRESS:XX.94.161 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.94.161 > STATUS:NORMAL,127605887595351923798765477788721654890 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.745883458E10 > /XX.180.36 > RACK:1b > RPC_ADDRESS:XX.123.112 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.123.112 > STATUS:NORMAL,1808575600 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:5.311963027E10 > /XX.163.36 > RACK:1c > RPC_ADDRESS:XX.163.180 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.163.180 > STATUS:NORMAL,92159807707754167187997289514579132865 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.7516755022E10 > /XX.31.234 > RACK:1d > RPC_ADDRESS:XX.192.159 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.192.159 > STATUS:NORMAL,99249023685273718510150927169407637270 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.7954372912E10 > /XX.197.209 > RACK:1d > RPC_ADDRESS:XX.66.205 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.66.205 > STATUS:NORMAL,141784319550391026443072753098378663700 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.4559013211E10 > /XX.177.177 > RACK:1d > RPC_ADDRESS:XX.65.57 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.65.57 > STATUS:NORMAL,14178431955039102644307275311465584410 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:3.6115572697E10 > /XX.20.9 > RACK:1c > RPC_ADDRESS:XX.33.229 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.33.229 > STATUS:NORMAL,49624511842636859255075463585608106435 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.1352367264E10 > /XX.231.121 > HOST_ID:9c765678-d058-4d85-a588-638ce10ff984 > RACK:1c > RPC_ADDRESS:XX.223.241 > RELEASE_VERSION:1.2.2 > INTERNAL_IP:XX.223.241 > STATUS:NORMAL,7089215977519551322153637656637080005 > NET_VERSION:7 > SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67 > SEVERITY:0.0 > DC:us-east > LOAD:5.0710624207E10 > Is this just a display bug in nodetool or this upgraded node really sees the > other ones as dead? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira