[
https://issues.apache.org/jira/browse/CASSANDRA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621118#comment-13621118
]
Brandon Williams commented on CASSANDRA-5332:
---------------------------------------------
Can you post gossipinfo from 1.1.10?
> 1.2.3 Nodetool reports 1.1.10 nodes as down
> -------------------------------------------
>
> Key: CASSANDRA-5332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5332
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.3
> Environment: Sun Java 6_u39
> Ubuntu 10.04.1
> Cassandra 1.2.3 and 1.1.10
> Reporter: Arya Goudarzi
>
> During exercise of a rolling upgrade from 1.1.10 to 1.2.3, I upgraded one
> node from 1.1.10 to 1.2.3 today. It appears that node running 1.2.3 reports
> nodes in 1.1.10 as Down in nodetool. This means that the nodes running 1.1.10
> see all other nodes including 1.2.3 as Up. Here is the ring and gossip from
> nodes with 1.1.10 for example. "231.121" is the upgraded node:
> Address DC Rack Status State Load
> Effective-Ownership Token
>
> 141784319550391026443072753098378663700
> XX.180.36 us-east 1b Up Normal 49.47 GB 25.00%
> 1808575600
> XX.231.121 us-east 1c Up Normal 47.08 GB 25.00%
> 7089215977519551322153637656637080005
> XX.177.177 us-east 1d Up Normal 33.64 GB 25.00%
> 14178431955039102644307275311465584410
> XX.7.148 us-east 1b Up Normal 41.27 GB 25.00%
> 42535295865117307932921825930779602030
> XX.20.9 us-east 1c Up Normal 38.51 GB 25.00%
> 49624511842636859255075463585608106435
> XX.86.255 us-east 1d Up Normal 34.78 GB 25.00%
> 56713727820156410577229101240436610840
> XX.63.230 us-east 1b Up Normal 38.11 GB 25.00%
> 85070591730234615865843651859750628460
> XX.163.36 us-east 1c Up Normal 44.25 GB 25.00%
> 92159807707754167187997289514579132865
> XX.31.234 us-east 1d Up Normal 44.66 GB 25.00%
> 99249023685273718510150927169407637270
> XX.132.169 us-east 1b Up Normal 44.2 GB 25.00%
> 127605887595351923798765477788721654890
> XX.71.63 us-east 1c Up Normal 38.74 GB 25.00%
> 134695103572871475120919115443550159295
> XX.197.209 us-east 1d Up Normal 41.5 GB 25.00%
> 141784319550391026443072753098378663700
> /XX.71.63
> RACK:1c
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.1598705272E10
> DC:us-east
> INTERNAL_IP:XX.194.92
> STATUS:NORMAL,134695103572871475120919115443550159295
> RPC_ADDRESS:XX.194.92
> RELEASE_VERSION:1.1.6
> /XX.86.255
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:3.734334162E10
> DC:us-east
> INTERNAL_IP:XX.6.195
> STATUS:NORMAL,56713727820156410577229101240436610840
> RPC_ADDRESS:XX.6.195
> RELEASE_VERSION:1.1.6
> /XX.7.148
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.4316975808E10
> DC:us-east
> INTERNAL_IP:XX.47.250
> STATUS:NORMAL,42535295865117307932921825930779602030
> RPC_ADDRESS:XX.47.250
> RELEASE_VERSION:1.1.6
> /XX.63.230
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.0918593305E10
> DC:us-east
> INTERNAL_IP:XX.89.127
> STATUS:NORMAL,85070591730234615865843651859750628460
> RPC_ADDRESS:XX.89.127
> RELEASE_VERSION:1.1.6
> /XX.132.169
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.745883458E10
> DC:us-east
> INTERNAL_IP:XX.94.161
> STATUS:NORMAL,127605887595351923798765477788721654890
> RPC_ADDRESS:XX.94.161
> RELEASE_VERSION:1.1.6
> /XX.180.36
> RACK:1b
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:5.311963027E10
> DC:us-east
> INTERNAL_IP:XX.123.112
> STATUS:NORMAL,1808575600
> RPC_ADDRESS:XX.123.112
> RELEASE_VERSION:1.1.6
> /XX.163.36
> RACK:1c
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.7516755022E10
> DC:us-east
> INTERNAL_IP:XX.163.180
> STATUS:NORMAL,92159807707754167187997289514579132865
> RPC_ADDRESS:XX.163.180
> RELEASE_VERSION:1.1.6
> /XX.31.234
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.7954372912E10
> DC:us-east
> INTERNAL_IP:XX.192.159
> STATUS:NORMAL,99249023685273718510150927169407637270
> RPC_ADDRESS:XX.192.159
> RELEASE_VERSION:1.1.6
> /XX.197.209
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.4558968005E10
> DC:us-east
> INTERNAL_IP:XX.66.205
> STATUS:NORMAL,141784319550391026443072753098378663700
> RPC_ADDRESS:XX.66.205
> RELEASE_VERSION:1.1.6
> /XX.177.177
> RACK:1d
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:3.6115572697E10
> DC:us-east
> INTERNAL_IP:XX.65.57
> STATUS:NORMAL,14178431955039102644307275311465584410
> RPC_ADDRESS:XX.65.57
> RELEASE_VERSION:1.1.6
> /XX.20.9
> RACK:1c
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> LOAD:4.1352503882E10
> DC:us-east
> INTERNAL_IP:XX.33.229
> STATUS:NORMAL,49624511842636859255075463585608106435
> RPC_ADDRESS:XX.33.229
> RELEASE_VERSION:1.1.6
> /XX.231.121
> RACK:1c
> SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0
> X4:9c765678-d058-4d85-a588-638ce10ff984
> X3:7
> DC:us-east
> INTERNAL_IP:XX.223.241
> RPC_ADDRESS:XX.223.241
> RELEASE_VERSION:1.2.2
> Now the nodetool on the 1.2.3 node shows all nodes as Down but itself.
> Gossipinfo looks good though:
> Datacenter: us-east
> ==========
> Replicas: 3
> Address Rack Status State Load Owns
> Token
>
> 56713727820156410577229101240436610840
> XX.132.169 1b Down Normal 44.2 GB 25.00%
> 127605887595351923798765477788721654890
> XX.7.148 1b Down Normal 41.27 GB 25.00%
> 42535295865117307932921825930779602030
> XX.180.36 1b Down Normal 49.47 GB 25.00%
> 1808575600
> XX.63.230 1b Down Normal 38.11 GB 25.00%
> 85070591730234615865843651859750628460
> XX.231.121 1c Up Normal 47.25 GB 25.00%
> 7089215977519551322153637656637080005
> XX.71.63 1c Down Normal 38.74 GB 25.00%
> 134695103572871475120919115443550159295
> XX.177.177 1d Down Normal 33.64 GB 25.00%
> 14178431955039102644307275311465584410
> XX.31.234 1d Down Normal 44.66 GB 25.00%
> 99249023685273718510150927169407637270
> XX.20.9 1c Down Normal 38.51 GB 25.00%
> 49624511842636859255075463585608106435
> XX.163.36 1c Down Normal 44.25 GB 25.00%
> 92159807707754167187997289514579132865
> XX.197.209 1d Down Normal 41.5 GB 25.00%
> 141784319550391026443072753098378663700
> XX.86.255 1d Down Normal 34.78 GB 25.00%
> 56713727820156410577229101240436610840
> /XX.71.63
> RACK:1c
> RPC_ADDRESS:XX.194.92
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.194.92
> STATUS:NORMAL,134695103572871475120919115443550159295
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.1598705272E10
> /XX.86.255
> RACK:1d
> RPC_ADDRESS:XX.6.195
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.6.195
> STATUS:NORMAL,56713727820156410577229101240436610840
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:3.7343205002E10
> /XX.7.148
> RACK:1b
> RPC_ADDRESS:XX.47.250
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.47.250
> STATUS:NORMAL,42535295865117307932921825930779602030
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.4316975808E10
> /XX.63.230
> RACK:1b
> RPC_ADDRESS:XX.89.127
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.89.127
> STATUS:NORMAL,85070591730234615865843651859750628460
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.0918456687E10
> /XX.132.169
> RACK:1b
> RPC_ADDRESS:XX.94.161
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.94.161
> STATUS:NORMAL,127605887595351923798765477788721654890
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.745883458E10
> /XX.180.36
> RACK:1b
> RPC_ADDRESS:XX.123.112
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.123.112
> STATUS:NORMAL,1808575600
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:5.311963027E10
> /XX.163.36
> RACK:1c
> RPC_ADDRESS:XX.163.180
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.163.180
> STATUS:NORMAL,92159807707754167187997289514579132865
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.7516755022E10
> /XX.31.234
> RACK:1d
> RPC_ADDRESS:XX.192.159
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.192.159
> STATUS:NORMAL,99249023685273718510150927169407637270
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.7954372912E10
> /XX.197.209
> RACK:1d
> RPC_ADDRESS:XX.66.205
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.66.205
> STATUS:NORMAL,141784319550391026443072753098378663700
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.4559013211E10
> /XX.177.177
> RACK:1d
> RPC_ADDRESS:XX.65.57
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.65.57
> STATUS:NORMAL,14178431955039102644307275311465584410
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:3.6115572697E10
> /XX.20.9
> RACK:1c
> RPC_ADDRESS:XX.33.229
> RELEASE_VERSION:1.1.6
> INTERNAL_IP:XX.33.229
> STATUS:NORMAL,49624511842636859255075463585608106435
> SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
> DC:us-east
> LOAD:4.1352367264E10
> /XX.231.121
> HOST_ID:9c765678-d058-4d85-a588-638ce10ff984
> RACK:1c
> RPC_ADDRESS:XX.223.241
> RELEASE_VERSION:1.2.2
> INTERNAL_IP:XX.223.241
> STATUS:NORMAL,7089215977519551322153637656637080005
> NET_VERSION:7
> SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67
> SEVERITY:0.0
> DC:us-east
> LOAD:5.0710624207E10
> Is this just a display bug in nodetool or this upgraded node really sees the
> other ones as dead?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
