[
https://issues.apache.org/jira/browse/CASSANDRA-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864657#comment-13864657
]
Michael Shuler commented on CASSANDRA-6554:
-------------------------------------------
Just the relevant portion:
{code}
$ sudo echo "== Begin ==" >> /var/log/cassandra/system.log ; nodetool status ;
sudo echo "== End ==" >> /var/log/cassandra/system.log
{code}
{code}
== Begin ==
TRACE [GossipTasks:1] 2014-01-07 20:29:59,731 Gossiper.java (line 133) My
heartbeat is now 438
TRACE [GossipTasks:1] 2014-01-07 20:29:59,732 Gossiper.java (line 398) Gossip
Digests are : /10.180.236.244:1389124753:2428 /10.91.137.10:1389126546:438
/10.182.208.161:1389124753:2265
TRACE [GossipTasks:1] 2014-01-07 20:29:59,732 Gossiper.java (line 530) Sending
a GossipDigestSyn to /10.182.208.161 ...
TRACE [GossipTasks:1] 2014-01-07 20:29:59,732 Gossiper.java (line 530) Sending
a GossipDigestSyn to /10.180.236.244 ...
TRACE [GossipTasks:1] 2014-01-07 20:29:59,733 Gossiper.java (line 598)
Performing status check ...
TRACE [GossipTasks:1] 2014-01-07 20:29:59,733 FailureDetector.java (line 215)
PHI for /10.180.236.244 : 0.5008780617528535
TRACE [GossipTasks:1] 2014-01-07 20:29:59,733 FailureDetector.java (line 215)
PHI for /10.182.208.161 : 0.6590114452533097
TRACE [GossipStage:1] 2014-01-07 20:29:59,734 GossipDigestAckVerbHandler.java
(line 41) Received a GossipDigestAckMessage from /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:29:59,735 GossipDigestAckVerbHandler.java
(line 52) Received ack with 1 digests and 2 states
TRACE [GossipStage:1] 2014-01-07 20:29:59,735 FailureDetector.java (line 189)
reporting /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:29:59,735 Gossiper.java (line 932)
/10.180.236.244local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:29:59,735 Gossiper.java (line 978) Updating
heartbeat state version to 2428 from 2428 for /10.180.236.244 ...
TRACE [GossipStage:1] 2014-01-07 20:29:59,749 Gossiper.java (line 932)
/10.182.208.161local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:29:59,749 Gossiper.java (line 978) Updating
heartbeat state version to 2269 from 2265 for /10.182.208.161 ...
TRACE [GossipStage:1] 2014-01-07 20:29:59,751 Gossiper.java (line 730) local
heartbeat version 438 greater than 433 for /10.91.137.10
TRACE [GossipStage:1] 2014-01-07 20:29:59,751 Gossiper.java (line 744) Adding
state SEVERITY: 0.0
TRACE [GossipStage:1] 2014-01-07 20:29:59,751 Gossiper.java (line 744) Adding
state SCHEMA: 9f0f4c3b-b568-3650-8ff7-39ae0d5be671
TRACE [GossipStage:1] 2014-01-07 20:29:59,752 GossipDigestAckVerbHandler.java
(line 84) Sending a GossipDigestAck2Message to /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:29:59,775 GossipDigestAckVerbHandler.java
(line 41) Received a GossipDigestAckMessage from /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,775 GossipDigestAckVerbHandler.java
(line 52) Received ack with 1 digests and 2 states
TRACE [GossipStage:1] 2014-01-07 20:29:59,775 Gossiper.java (line 932)
/10.180.236.244local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:29:59,775 Gossiper.java (line 952) Ignoring
remote version 2431 <= 2431 for /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,776 Gossiper.java (line 932)
/10.182.208.161local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:29:59,776 Gossiper.java (line 952) Ignoring
remote version 2269 <= 2270 for /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:29:59,790 Gossiper.java (line 730) local
heartbeat version 438 greater than 433 for /10.91.137.10
TRACE [GossipStage:1] 2014-01-07 20:29:59,791 Gossiper.java (line 744) Adding
state SEVERITY: 0.0
TRACE [GossipStage:1] 2014-01-07 20:29:59,791 Gossiper.java (line 744) Adding
state SCHEMA: 9f0f4c3b-b568-3650-8ff7-39ae0d5be671
TRACE [GossipStage:1] 2014-01-07 20:29:59,791 GossipDigestAckVerbHandler.java
(line 84) Sending a GossipDigestAck2Message to /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,971 GossipDigestSynVerbHandler.java
(line 40) Received a GossipDigestSynMessage from /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,971 GossipDigestSynVerbHandler.java
(line 71) Gossip syn digests are : /10.182.208.161:1389124753:2269
/10.91.137.10:1389126546:438 /10.180.236.244:1389124753:2432
TRACE [GossipStage:1] 2014-01-07 20:29:59,971 Gossiper.java (line 744) Adding
state SCHEMA: 9f0f4c3b-b568-3650-8ff7-39ae0d5be671
TRACE [GossipStage:1] 2014-01-07 20:29:59,972 Gossiper.java (line 744) Adding
state SCHEMA: 59adb24e-f3cd-3e02-97f0-5b395827453f
TRACE [GossipStage:1] 2014-01-07 20:29:59,972 GossipDigestSynVerbHandler.java
(line 79) sending 1 digests and 2 deltas
TRACE [GossipStage:1] 2014-01-07 20:29:59,972 GossipDigestSynVerbHandler.java
(line 84) Sending a GossipDigestAckMessage to /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,986 GossipDigestAck2VerbHandler.java
(line 38) Received a GossipDigestAck2Message from /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,987 FailureDetector.java (line 189)
reporting /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:29:59,987 Gossiper.java (line 932)
/10.180.236.244local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:29:59,987 Gossiper.java (line 978) Updating
heartbeat state version to 2432 from 2428 for /10.180.236.244 ...
TRACE [GossipStage:1] 2014-01-07 20:30:00,594 GossipDigestSynVerbHandler.java
(line 40) Received a GossipDigestSynMessage from /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,594 GossipDigestSynVerbHandler.java
(line 71) Gossip syn digests are : /10.182.208.161:1389124753:2276
/10.91.137.10:1389126546:438 /10.180.236.244:1389124753:2432
TRACE [GossipStage:1] 2014-01-07 20:30:00,594 Gossiper.java (line 744) Adding
state SEVERITY: 0.0
TRACE [GossipStage:1] 2014-01-07 20:30:00,594 Gossiper.java (line 744) Adding
state SCHEMA: 9f0f4c3b-b568-3650-8ff7-39ae0d5be671
TRACE [GossipStage:1] 2014-01-07 20:30:00,595 GossipDigestSynVerbHandler.java
(line 79) sending 1 digests and 1 deltas
TRACE [GossipStage:1] 2014-01-07 20:30:00,595 GossipDigestSynVerbHandler.java
(line 84) Sending a GossipDigestAckMessage to /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,607 GossipDigestAck2VerbHandler.java
(line 38) Received a GossipDigestAck2Message from /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,607 FailureDetector.java (line 189)
reporting /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,607 Gossiper.java (line 932)
/10.182.208.161local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:30:00,607 Gossiper.java (line 978) Updating
heartbeat state version to 2276 from 2269 for /10.182.208.161 ...
TRACE [GossipTasks:1] 2014-01-07 20:30:00,750 Gossiper.java (line 133) My
heartbeat is now 445
TRACE [GossipTasks:1] 2014-01-07 20:30:00,751 Gossiper.java (line 398) Gossip
Digests are : /10.91.137.10:1389126546:445 /10.180.236.244:1389124753:2432
/10.182.208.161:1389124753:2276
TRACE [GossipTasks:1] 2014-01-07 20:30:00,751 Gossiper.java (line 530) Sending
a GossipDigestSyn to /10.182.208.161 ...
TRACE [GossipTasks:1] 2014-01-07 20:30:00,751 Gossiper.java (line 530) Sending
a GossipDigestSyn to /10.180.236.244 ...
TRACE [GossipTasks:1] 2014-01-07 20:30:00,752 Gossiper.java (line 598)
Performing status check ...
TRACE [GossipTasks:1] 2014-01-07 20:30:00,752 FailureDetector.java (line 215)
PHI for /10.180.236.244 : 0.503756137674087
TRACE [GossipTasks:1] 2014-01-07 20:30:00,753 FailureDetector.java (line 215)
PHI for /10.182.208.161 : 0.09675013184828929
TRACE [GossipStage:1] 2014-01-07 20:30:00,753 GossipDigestAckVerbHandler.java
(line 41) Received a GossipDigestAckMessage from /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,754 GossipDigestAckVerbHandler.java
(line 52) Received ack with 1 digests and 1 states
TRACE [GossipStage:1] 2014-01-07 20:30:00,754 Gossiper.java (line 932)
/10.182.208.161local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:30:00,754 Gossiper.java (line 978) Updating
heartbeat state version to 2276 from 2276 for /10.182.208.161 ...
TRACE [GossipStage:1] 2014-01-07 20:30:00,762 Gossiper.java (line 730) local
heartbeat version 445 greater than 444 for /10.91.137.10
TRACE [GossipStage:1] 2014-01-07 20:30:00,763 GossipDigestAckVerbHandler.java
(line 84) Sending a GossipDigestAck2Message to /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,763 GossipDigestAckVerbHandler.java
(line 41) Received a GossipDigestAckMessage from /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:30:00,763 GossipDigestAckVerbHandler.java
(line 52) Received ack with 2 digests and 1 states
TRACE [GossipStage:1] 2014-01-07 20:30:00,764 Gossiper.java (line 932)
/10.180.236.244local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:30:00,764 Gossiper.java (line 978) Updating
heartbeat state version to 2432 from 2432 for /10.180.236.244 ...
TRACE [GossipStage:1] 2014-01-07 20:30:00,765 Gossiper.java (line 730) local
heartbeat version 445 greater than 440 for /10.91.137.10
TRACE [GossipStage:1] 2014-01-07 20:30:00,765 Gossiper.java (line 744) Adding
state SEVERITY: 0.0
TRACE [GossipStage:1] 2014-01-07 20:30:00,765 Gossiper.java (line 744) Adding
state SCHEMA: 9f0f4c3b-b568-3650-8ff7-39ae0d5be671
TRACE [GossipStage:1] 2014-01-07 20:30:00,766 Gossiper.java (line 730) local
heartbeat version 2276 greater than 2272 for /10.182.208.161
TRACE [GossipStage:1] 2014-01-07 20:30:00,766 Gossiper.java (line 744) Adding
state SEVERITY: 0.0
TRACE [GossipStage:1] 2014-01-07 20:30:00,766 Gossiper.java (line 744) Adding
state SCHEMA: 59adb24e-f3cd-3e02-97f0-5b395827453f
TRACE [GossipStage:1] 2014-01-07 20:30:00,766 GossipDigestAckVerbHandler.java
(line 84) Sending a GossipDigestAck2Message to /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:30:00,961 GossipDigestSynVerbHandler.java
(line 40) Received a GossipDigestSynMessage from /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:30:00,962 GossipDigestSynVerbHandler.java
(line 71) Gossip syn digests are : /10.91.137.10:1389126546:445
/10.180.236.244:1389124753:2441 /10.182.208.161:1389124753:2279
TRACE [GossipStage:1] 2014-01-07 20:30:00,962 GossipDigestSynVerbHandler.java
(line 79) sending 1 digests and 0 deltas
TRACE [GossipStage:1] 2014-01-07 20:30:00,962 GossipDigestSynVerbHandler.java
(line 84) Sending a GossipDigestAckMessage to /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:30:00,964 GossipDigestAck2VerbHandler.java
(line 38) Received a GossipDigestAck2Message from /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:30:00,964 FailureDetector.java (line 189)
reporting /10.180.236.244
TRACE [GossipStage:1] 2014-01-07 20:30:00,964 Gossiper.java (line 932)
/10.180.236.244local generation 1389124753, remote generation 1389124753
TRACE [GossipStage:1] 2014-01-07 20:30:00,964 Gossiper.java (line 978) Updating
heartbeat state version to 2441 from 2432 for /10.180.236.244 ...
== End ==
{code}
> During upgrade from 1.2 -> 2.0, upgraded node sees other nodes as Down
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-6554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6554
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: EC2 Ubuntu Precise 12.04
> Oracle JRE 1.7_25
> C* 1.2.13 upgrade to 2.0.4
> Reporter: Michael Shuler
> Attachments: 6554_trace_system.log
>
>
> During an upgrade from 1.2.13 to 2.0.3/2.0.4, the upgraded node sees the
> remaining nodes of the cluster as Down.
> {code}
> automaton@ip-10-139-1-113:~$ nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Owns Host ID
> Token Rack
> UN 10.139.1.113 98.94 MB 33.3% 33b1cd06-e17b-4332-8066-0c6c401e0cf3
> -9223372036854775808 rack1
> DN 10.139.11.168 97.51 MB 33.3% ec97c163-8f2d-4019-a3d1-55df5e4037d4
> -3074457345618258603 rack1
> DN 10.238.221.115 97.34 MB 33.3% 73a76d3f-73ef-481d-b603-0833c0ff80c2
> 3074457345618258602 rack1
> automaton@ip-10-139-1-113:~$ nodetool gossipinfo
> /10.238.221.115
> SEVERITY:0.0
> RPC_ADDRESS:0.0.0.0
> DC:datacenter1
> RELEASE_VERSION:1.2.13
> LOAD:1.02066255E8
> STATUS:NORMAL,3074457345618258602
> SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
> NET_VERSION:6
> RACK:rack1
> HOST_ID:73a76d3f-73ef-481d-b603-0833c0ff80c2
> /10.139.1.113
> RPC_ADDRESS:0.0.0.0
> SEVERITY:0.0
> DC:datacenter1
> RELEASE_VERSION:2.0.4
> LOAD:1.03750451E8
> STATUS:NORMAL,-9223372036854775808
> SCHEMA:dfafb212-5b8f-31cb-a80b-2ba58fcef73d
> NET_VERSION:7
> RACK:rack1
> HOST_ID:33b1cd06-e17b-4332-8066-0c6c401e0cf3
> /10.139.11.168
> SEVERITY:0.0
> RPC_ADDRESS:0.0.0.0
> DC:datacenter1
> RELEASE_VERSION:1.2.13
> LOAD:1.02245066E8
> STATUS:NORMAL,-3074457345618258603
> SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
> NET_VERSION:6
> RACK:rack1
> HOST_ID:ec97c163-8f2d-4019-a3d1-55df5e4037d4
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
