[ https://issues.apache.org/jira/browse/CASSANDRA-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146124#comment-13146124 ]
Jonas Borgström commented on CASSANDRA-3466: -------------------------------------------- > Do you observe the same behavior if both nodes start life as 1.x? No. Hinted handoff seems to work well if I either start with 1.0.2 nodes or restart all nodes once all nodes have been upgraded to 1.0.2. Probably something in the gossip state confusing hinted hand-offs. > Is node2 down long enough for node1 failure detector to notice? (i.e. shows > as DOWN in nodetool ring) Yes. This from the node1 log: {code} Node2 is shut down INFO 19:39:08,331 InetAddress /127.0.0.2 is now dead. Hint is triggered using set bar[x][x]='x' INFO 19:40:52,126 Node /127.0.0.2 has restarted, now UP INFO 19:40:52,127 InetAddress /127.0.0.2 is now UP INFO 19:40:52,127 Node /127.0.0.2 state jump to normal Nothing happens, hint stays in HintsColumnFamily forever {code} Calling deliverHints() using JMX also does not seem to work. Restarting node1 will though... > Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2 > -------------------------------------------------------------------- > > Key: CASSANDRA-3466 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3466 > Project: Cassandra > Issue Type: Bug > Affects Versions: 1.0.2 > Reporter: Jonas Borgström > Labels: hintedhandoff > > While testing rolling upgrades from 0.8.7 to 1.0.2 on a test cluster I've > noticed that hinted hand-off didn't always work properly. Hints generated on > an upgraded node does not seem to be delivered to other newly upgraded nodes > once they rejoin the ring. They only way I've found to get a node to deliver > its hints is to restart it. > Here's some steps to reproduce this issue: > 1. Install cassandra 0.8.7 on node1 and node2 using default settings. > 2. Create keyspace foo with {replication_factor: 2}. Create column family bar > 3. Shutdown node2 > 4. Insert data into bar and verify that HintsColumnFamily on node2 contains > hints > 5. Start node2 and verify that hinted handoff is performed and > HintsColumnFamily becomes empty again. > 6. Upgrade and restart node1 > 7. Shutdown node2 > 8. Insert data into bar and verify that HintsColumnFamily on node2 contains > hints > 9. Upgrade and start node2 > 10. Notice that hinted handoff is *not* performed when "node2" comes back. > (Only if node1 is restarted) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira