[ https://issues.apache.org/jira/browse/CASSANDRA-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165477#comment-16165477 ]
Aleksey Yeschenko commented on CASSANDRA-12872: ----------------------------------------------- A minimal dtest reproducing the issue [here|https://github.com/iamaleksey/cassandra-dtest/commits/12872]. CASSANDRA-13794 changes make it pass, but unfortunately the bug itself is still here - it's just masked by min(limit, 16) optimisation in CASSANDRA-13794 branch. [~blerer] I'm close-ish to completion here. Mind if I take over? It's a bit time-sensitive. If you do mind and/or have something in progress, feel free to take it back. > Paging reads and limit reads are missing some data > -------------------------------------------------- > > Key: CASSANDRA-12872 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12872 > Project: Cassandra > Issue Type: Bug > Components: Coordination > Reporter: Bhaskar Muppana > Assignee: Benjamin Lerer > Priority: Critical > Labels: Correctness > Fix For: 3.0.x > > Attachments: limiterr-reproduce.sh > > > We are seeing an issue with paging reads missing some small number of columns > when we do paging/limit reads. We get this on a single DC cluster itself when > both reads and writes are happening with QUORUM. Paging/limit reads see this > issue. I have attached the ccm based script which reproduces the problem. > * Keyspace RF - 3 > * Table (id int, course text, marks int, primary key(id, course)) > * replicas for partition key 1 - r1, r2 and r3 > * insert (1, '1', 1) , (1, '2', 2), (1, '3', 3), (1, '4', 4), (1, '5', 5) > - succeeded on all 3 replicas > * insert (1, '6', 6) succeeded on r1 and r3, failed on r2 > * delete (1, '2'), (1, '3'), (1, '4'), (1, '5') succeeded on r1 and r2, > failed on r3 > * insert (1, '7', 7) succeeded on r1 and r2, failed on r3 > Local data on 3 nodes looks like as below now > r1: (1, '1', 1), tombstone(2-5 records), (1, '6', 6), (1, '7', 7) > r2: (1, '1', 1), tombstone(2-5 records), (1, '7', 7) > r3: (1, '1', 1), (1, '2', 2), (1, '3', 3), (1, '4', 4), (1, '5', 5), (1, > '6', 6) > If we do a paging read with page_size 2, and if it gets data from r2 and r3, > then it will only get the data (1, '1', 1) and (1, '7', 7) skipping record 6. > This problem would happen if the same query is not doing paging but limit set > to 2 records. > Resolution code for reads works same for paging queries and normal queries. > Co-ordinator shouldn't respond back to client with records/columns that it > didn't have complete visibility on all required replicas (in this case 2 > replicas). In above case, it is sending back record (1, '7', 7) back to > client, but its visibility on r3 is limited up to (1, '2', 2) and it is > relying on just r2 data to assume (1, '6', 6) doesn't exist, which is wrong. > End of the resolution all it can conclusively say any thing about is (1, '1', > and the other one is that we and and and and and and the and the and the > and d and the other is and 1), which exists and (1, '2', 2), which is deleted. > Ideally we should have different resolution implementation for paging/limit > queries. > We could reproduce this on 2.0.17, 2.1.16 and 3.0.9. > Seems like 3.0.9 we have ShortReadProtection transformation on list queries. > I assume that is to protect against the cases like above. But, we can > reproduce the issue in 3.0.9 as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org