Hi, We are using Cassandra 2.0.8, with replication factor of 3. We are seeing a scenario where some of the rows in the table reappears even after they are deleted. We have seen this in Prod 3 times in last 1 week and coincidentally all 3 times on the same partition. We have confirmed that nodes didn't go down (they were all started on Oct 28). I do not see any resource issues (CPU is around 5%, memory < 10%, iostats looks normal etc) on any nodes.
The only possible explanation we have so far is, one node that has data for affected partition, being browned out for at least 15 minutes (our GC grace is 15 minutes) and came back up but when it came back up, there were some rows in that partition that were deleted on other two nodes. However I am not sure how to prove this is what is happening. I am not sure if this is a bug in Cassandra (unlikely since we have not seen this issue in last 8 months until this week). Any pointers as to how to find the root cause would be greatly appreciated. Thanks in advance. Praveen
