[ https://issues.apache.org/jira/browse/CASSANDRA-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aleksey Yeschenko updated CASSANDRA-12872: ------------------------------------------ Summary: Fix short read protection when more than one row is missing (was: Paging reads and limit reads are missing some data) > Fix short read protection when more than one row is missing > ----------------------------------------------------------- > > Key: CASSANDRA-12872 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12872 > Project: Cassandra > Issue Type: Bug > Components: Coordination > Reporter: Bhaskar Muppana > Assignee: Aleksey Yeschenko > Priority: Critical > Labels: Correctness > Fix For: 3.0.15, 3.11.1 > > Attachments: limiterr-reproduce.sh > > > We are seeing an issue with paging reads missing some small number of columns > when we do paging/limit reads. We get this on a single DC cluster itself when > both reads and writes are happening with QUORUM. Paging/limit reads see this > issue. I have attached the ccm based script which reproduces the problem. > * Keyspace RF - 3 > * Table (id int, course text, marks int, primary key(id, course)) > * replicas for partition key 1 - r1, r2 and r3 > * insert (1, '1', 1) , (1, '2', 2), (1, '3', 3), (1, '4', 4), (1, '5', 5) > - succeeded on all 3 replicas > * insert (1, '6', 6) succeeded on r1 and r3, failed on r2 > * delete (1, '2'), (1, '3'), (1, '4'), (1, '5') succeeded on r1 and r2, > failed on r3 > * insert (1, '7', 7) succeeded on r1 and r2, failed on r3 > Local data on 3 nodes looks like as below now > r1: (1, '1', 1), tombstone(2-5 records), (1, '6', 6), (1, '7', 7) > r2: (1, '1', 1), tombstone(2-5 records), (1, '7', 7) > r3: (1, '1', 1), (1, '2', 2), (1, '3', 3), (1, '4', 4), (1, '5', 5), (1, > '6', 6) > If we do a paging read with page_size 2, and if it gets data from r2 and r3, > then it will only get the data (1, '1', 1) and (1, '7', 7) skipping record 6. > This problem would happen if the same query is not doing paging but limit set > to 2 records. > Resolution code for reads works same for paging queries and normal queries. > Co-ordinator shouldn't respond back to client with records/columns that it > didn't have complete visibility on all required replicas (in this case 2 > replicas). In above case, it is sending back record (1, '7', 7) back to > client, but its visibility on r3 is limited up to (1, '2', 2) and it is > relying on just r2 data to assume (1, '6', 6) doesn't exist, which is wrong. > End of the resolution all it can conclusively say any thing about is (1, '1', > and the other one is that we and and and and and and the and the and the > and d and the other is and 1), which exists and (1, '2', 2), which is deleted. > Ideally we should have different resolution implementation for paging/limit > queries. > We could reproduce this on 2.0.17, 2.1.16 and 3.0.9. > Seems like 3.0.9 we have ShortReadProtection transformation on list queries. > I assume that is to protect against the cases like above. But, we can > reproduce the issue in 3.0.9 as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org