[
https://issues.apache.org/jira/browse/CASSANDRA-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonas Borgström updated CASSANDRA-3466:
---------------------------------------
Attachment: CASSANDRA-3466-2.tar.gz
I've unfortunately managed to reproduce this AssertionError even with the
CASSANDRA-3446 patches and the HintedHandOffManager.java modification.
I used the same two node setup as before initialized with:
{code}
create keyspace foo with placement_strategy =
'org.apache.cassandra.locator.SimpleStrategy' and strategy_options =
{replication_factor:2};
use foo;
create column family bar with comparator=UTF8Type and
key_validation_class=UTF8Type;
{code}
# Shutdown node2 and wait until it's detected as down
trigger hints on node1 (using random values):
{code}
use foo;
set bar[0][0]='1';
set bar[2][3]='1';
set bar[3][3]='1';
{code}
Verify that `list HintsColumnFamily` looks OK
# Start node2 and wait for hints delivery
Verify that `list HintsColumnFamily` is empty after hints delivery
After repeating step 1-2 a few times (sometimes once is enough)
"HintsColumnFamily" will become "corrupt" after hints delivery instead of
empty. And if node1 is restarted after that it will trigger the AssertionError.
The attached file contains DEBUG-log files, data directories and
conf-directories from one test where HintsColumnFamily ended up looking like
this:
{code}
[default@system] list HintsColumnFamily;
Using default limit of 100
-------------------
RowKey: 1b226da5af66854850abdcc6ab4ce9c6
=> (super_column=1f2860e00b9811e10000fe8ebeead9ff,
(column=6b6579, value=33, timestamp=1320928378352, ttl=864000))
1 Row Returned.
Elapsed time: 6 msec(s).
{code}
> Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
> --------------------------------------------------------------------
>
> Key: CASSANDRA-3466
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3466
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Jonas Borgström
> Assignee: Brandon Williams
> Labels: hintedhandoff
> Fix For: 1.0.3
>
> Attachments: CASSANDRA-3466-2.tar.gz, CASSANDRA-3466.tar.gz
>
>
> While testing rolling upgrades from 0.8.7 to 1.0.2 on a test cluster I've
> noticed that hinted hand-off didn't always work properly. Hints generated on
> an upgraded node does not seem to be delivered to other newly upgraded nodes
> once they rejoin the ring. They only way I've found to get a node to deliver
> its hints is to restart it.
> Here's some steps to reproduce this issue:
> 1. Install cassandra 0.8.7 on node1 and node2 using default settings.
> 2. Create keyspace foo with {replication_factor: 2}. Create column family bar
> 3. Shutdown node2
> 4. Insert data into bar and verify that HintsColumnFamily on node2 contains
> hints
> 5. Start node2 and verify that hinted handoff is performed and
> HintsColumnFamily becomes empty again.
> 6. Upgrade and restart node1
> 7. Shutdown node2
> 8. Insert data into bar and verify that HintsColumnFamily on node2 contains
> hints
> 9. Upgrade and start node2
> 10. Notice that hinted handoff is *not* performed when "node2" comes back.
> (Only if node1 is restarted)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
