I’ve confirmed that the inconsistency issues disappeared after repair finished.
Anything changed with repair in 3.11.1? One difference I noticed is that the validation step during repair could turn down the node upon large tables, which never happen in 3.10. I had to throttle validation requests to let it pass. Also I switched back to -pr instead of incremental repair which is a resource killer and often hangs for the first node to be repaired. To address the inconsistency issue, I could do Write All and Read One by giving up availability and stop running repair. Any comments on that? I guess I could also try downgrade c* version but will data file be a problem? Li > On Jul 3, 2018, at 15:53, kurt greaves <k...@instaclustr.com> wrote: > > Shouldn't happen. Any chance you could trace the queries, or have you been > able to reproduce it? Also, what version of Cassandra? > >> On Wed., 4 Jul. 2018, 06:41 Visa, <liguilin2...@gmail.com> wrote: >> Hi all, >> >> We recently experienced an unexpected behavior with C* consistency. >> >> For example, a table t consists of 4 columns - pk , a, b and c. We perform >> Quorum write and then Quorum read (RF=3 / LCS compaction). >> >> The consistency seems to break while repairing is running(repair -pr). >> >> Say, a record already exists in t like >> pk=1, a=1, b=1, c=1 >> >> While repair is not running >> >> Quorum Write: >> update t set c = 2 where pk=1 >> >> Quorum Read: >> select pk,a,b,c from t where pk=1 limit 1 >> >> Returns: (1, 1, 1, 2) as expected. >> >> But if we do it while repair is running, >> >> Quorum Write: >> update t set c=3 where pk=1 >> >> Quorum Read, however, returns (1, null, null, 3) w/o values of a and b. >> >> After repair is done, then the same Quorum Read returns the right values >> (1,1,1,3). >> >> It does not happen to every row in t. The impacted rows are like 40 out of >> 300 millions. But still how the consistency gets broken here? >> >> Thanks for your attention! >> >> Li >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >>
