[
https://issues.apache.org/jira/browse/CASSANDRA-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612485#comment-14612485
]
Hazel Bobrins commented on CASSANDRA-9715:
------------------------------------------
Updated - sorry seem to have posted blank before.
> Secondary index out of sync
> ---------------------------
>
> Key: CASSANDRA-9715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: RHEL 6.2 2.6.32-220.13.1.el6.x86_64
> Reporter: Hazel Bobrins
>
> On 2.0.15 ( we moved from 2.08 hoping this problem would go away) we am
> seeing intermittent issues where a secondary index is getting out of sync.
> Set up is a 6 node cluster with 3 data centers, two nodes in each and with a
> RF of 2 in each data centre.
> So far I have been unable to reproduce this synthetically but have seen
> multiple instances across all nodes within the cluster.
> Data set is very small ~40K keys and >100MB of data. We add maybe 1000
> records a day, delete ~500 and update ~200. Not a very write based system.
> Reads we can push out to ~2000/sec.
> Writes are done at CL ALL and reads at ONE
> All examples so far have been triggered when a record has been deleted and
> then other added with the same index cardinality; I think it has also always
> been the last record in the set which was deleted before the addition.
> On a flushed keyspace a sstable2json export of the primary index shows all
> records correctly, however, an export of the secondary index is missing the
> records.
> nodetool rebuild_index does not resolve the problem
> Nether does a compact or repair
> A select on the primary key at CL ALL also has no impact
> However, a select at CL ALL on the secondary index does resolve the problem.
> There is currently a none critical record which is out of the index on one of
> our nodes. If another key is added with the same index cardinality it is
> added to the index correctly. If this is then removed it once again returns
> empty.
> We have checked all the obvious OS bits and confirmed our time sync (ntp
> based).
> At DEBUG level we see nothing obvious wrong when adding/removing keys to the
> above broken entry.
> Due to the very intermittent nature of this problem is been impossible so far
> to gather any DEBUG logs of it failing; we have also been unsuccessful so far
> in reproducing this in out QA.
> I know this is not much to go on, if there is anything we can provide to help
> expand what might be the issue please let me know and we'll provide it asap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
