Trouble After Changing Replication Factor

[Isaeed Mohanna]

Hi
We had a cluster with 3 Nodes with Replication Factor 2 and we were using read 
with consistency Level One.
We recently added a 4th node and changed the replication factor to 3, once this 
was done apps reading from DB with CL1 would receive an empty record, Looking 
around I was surprised to learn that upon changing the replication factor if 
the read request is sent to a node the should own the record according to the 
new replication factor while it still doesn't have it yet then an empty record 
will be returned because of CL1, the record will be written to that node after 
the repair operation is over.
We ran the repair operation which took days in our case (we had to change apps 
to CL2 to avoid serious data inconsistencies).
Now the repair operations are over and if I revert to CL1 we are still getting 
errors that records do not exist in DB while they do, using CL2 again it works 
fine.
Any ideas what I am missing?
Is there a way to validate that the repairs task has actually done what is 
needed and that the data is actually now replicated RF3 ?
Could it it be a Cassandra Driver issue? Since if I issue the request in cqlsh 
I do get the record but I cannot know if I am hitting the replica that doesn't 
hold the record
Thanks for your help

Re: moving from 4.0-alpha4 to 4.0.1

[Attila Wind]

Thank you Paulo for your detailed answer!
I was not monitoring NEWS.txt in the Git repo so far but that file 
definitely has info I was looking for.


cheers

Attila Wind

http://www.linkedin.com/in/attilaw 
Mobile: +49 176 43556932


09.10.2021 15:07 keltezéssel, Paulo Motta írta:

Hi Attila,

Minor version upgrades are generally fine to do in-place, unless 
otherwise specified on NEWS.txt 
> 
for the specific versions you're upgrading. Cassandra is designed with 
this goal in mind, and potentially disruptive changes can only be 
introduced in major versions, which require a little more care during 
the upgrade process.


It's definitely safe to do an in-place one-node-at-a-time upgrade for 
minor versions in the same major series (ie. 4.0-alpha to 4.0.1). 
Nevertheless it doesn't hurt to take a global snapshot "just-in-case", 
so you can rollback in case you run into an unexpected issue, but this 
is just extra safety and not strictly required.


Unfortunately there's no official upgrade guide yet, this is something 
the community is working on to provide soon, but you can find some 
unofficial ones with a quick google search.


Major upgrades are also designed to be harmless, but a little bit more 
preparation is required to ensure a smooth ride due to potentially 
non-compatible changes. I've written an upgrade guide sometime ago 
which can be useful to prepare for a major upgrade, but can also apply 
to minor upgrades as well to ensure extra safety during the process: 
http://monkeys.chaordic.com.br/operation/2014/04/11/zero-downtime-cassandra-upgrade.html 



Cheers and good luck!

Paulo

Em sáb., 9 de out. de 2021 às 06:56, Attila Wind 
 escreveu:


Hi all,

I have 2 quick questions

1. We have a cluster running 4.0-alpha4. Now 4.0.1 is out and
obviously it would make lots of sense to switch to this version.
Does anyone know if we can do it simply "in place"? I mean we just
ugrade the software and restart? Or it would not work / would be
dangerous due to some storage layer incompatibilities or other
risk factors? So better to run a (usual) data migration process..?

2. Actually the above brought the more generic question: is the
community maintaining any kind of guide/readme/whatever one can
use to find answer for similar questions? As a user I see the
changelog and that's cool but that is not providing obvious
answers (of course). So I mean some sort of migration hints/guide.

thanks!

-- 
Attila Wind


http://www.linkedin.com/in/attilaw

Mobile: +49 176 43556932