Hi, You could try to run full repair over short subrange containing data missing from one replica. It should take just couple of minutes and will prove if your repair failed to finish
Dmitrii Saprykin On Tue, Oct 12, 2021 at 7:54 AM Bowen Song <bo...@bso.ng> wrote: > I see. In that case, I suspect the repair wasn't fully successful. Try > repair the new joined node again, and make sure it actually finishes > successfully. > On 12/10/2021 12:23, Isaeed Mohanna wrote: > > Hi > > Yes I am sacrificing consistency to gain higher availability and faster > speed, but my problem is not with newly inserted data that is not there for > a very short period of time, my problem is the data that was there before > the RF change, still do not exist in all replicas even after repair. > > It looks like my cluster configuration is RF3 but the data itself is still > using RF2 and when the data is requested from the 3rd (new) replica, it > is not there and an empty record is returned with read CL1. > > What can I do to force this data to be synced to all replicas as it > should? So read CL1 request will actually return a correct result? > > > > Thanks > > > > *From:* Bowen Song <bo...@bso.ng> <bo...@bso.ng> > *Sent:* Monday, October 11, 2021 5:13 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Trouble After Changing Replication Factor > > > > You have RF=3 and both read & write CL=1, which means you are asking > Cassandra to give up strong consistency in order to gain higher > availability and perhaps slight faster speed, and that's what you get. If > you want to have strong consistency, you will need to make sure (read CL + > write CL) > RF. > > On 10/10/2021 11:55, Isaeed Mohanna wrote: > > Hi > > We had a cluster with 3 Nodes with Replication Factor 2 and we were using > read with consistency Level One. > > We recently added a 4th node and changed the replication factor to 3, > once this was done apps reading from DB with CL1 would receive an empty > record, Looking around I was surprised to learn that upon changing the > replication factor if the read request is sent to a node the should own the > record according to the new replication factor while it still doesn’t have > it yet then an empty record will be returned because of CL1, the record > will be written to that node after the repair operation is over. > > We ran the repair operation which took days in our case (we had to change > apps to CL2 to avoid serious data inconsistencies). > > Now the repair operations are over and if I revert to CL1 we are still > getting errors that records do not exist in DB while they do, using CL2 > again it works fine. > > Any ideas what I am missing? > > Is there a way to validate that the repairs task has actually done what is > needed and that the data is actually now replicated RF3 ? > > Could it it be a Cassandra Driver issue? Since if I issue the request in > cqlsh I do get the record but I cannot know if I am hitting the replica > that doesn’t hold the record > > Thanks for your help > >
