Without knowing the cause of the issue, it's hard to tell what are the
correct steps to recover from it. I would recommend you have a look at
the logs and figure out what was the cause of the issue, and then make a
recovery plan and also put preventive measure in place to stop it from
happening again.
Also, I'm not comfortable with the idea of manually creating data
directories for Cassandra unless I know the system well, as this may
lead to filesystem ownership and permission issues. The involvement of
security tools like AppArmor or SELinux may make it even more complicated.
On 18/01/2024 15:46, ENES ATABERK wrote:
ok thank you!
What do you think about the following approach:
1. creating empty correct table id directories in linux filesystem
with respect to the system_schema.tables id column
2. importing data with nodetool import from incorrect directory
3. removing the incorrect directory afterwards
------------------------------------------------------------------------
*From:* Bowen Song via user <user@cassandra.apache.org>
*Sent:* Thursday, January 18, 2024 5:34:57 PM
*To:* user@cassandra.apache.org
*Cc:* Bowen Song
*Subject:* COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re:
system_schema.tables id and table uuid on disk mismatch
I know dropping a table and then creating a new table with the same
name can lead to that result, which is expected. If that wasn't what
happened, it may be a bug in Cassandra. If you can reproduce the
behaviour, you should raise a Jira ticket for it.
On 18/01/2024 14:44, ENES ATABERK wrote:
It has same mismatch id in all nodes not just one node.
------------------------------------------------------------------------
*From:* Bowen Song via user <user@cassandra.apache.org>
*Sent:* Thursday, January 18, 2024 3:18:11 PM
*To:* user@cassandra.apache.org
*Cc:* Bowen Song
*Subject:* COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and
table uuid on disk mismatch
Was the table ID mismatching only on one node or all nodes?
Mismatching on one node is usually the result of a racing condition,
but on all nodes isn't. The solution I mentioned earlier only applies
to the one node situation.
On 18/01/2024 13:14, ENES ATABERK wrote:
Hi all,
Thanks for your responses.
The version is Cassandra 4.1.3
After I restarted all the nodes one-by-one cassandra created
corrected-id folder and keep the incorrect one as you said.
But then I cannot see the data from cqlsh it gives me no result.
After i have imported the data from incorrect-id-folder i see the data
nodetool import keyspace_name table_name
/full_path_of_old(incorrect)_folder.
now my questions are like:
First question; before i have restarted the nodes how can i search
the data although there is a mismatch between system_schema.tables
and actual directories in all nodes.
Second one; is nodetool import a safe way to load data from
incorrect folder on a write heavy system. Because I cannot be sure
if I miss any data during the import operation. Or do i need to run
a repair for those tables instead of import? In my opinion that
may not work because i cannot see any data before nodetool import.
Thanks again.
------------------------------------------------------------------------
*From:* Bowen Song via user <user@cassandra.apache.org>
*Sent:* Thursday, January 18, 2024 1:17:11 PM
*To:* user@cassandra.apache.org
*Cc:* Bowen Song
*Subject:* COMMERCIAL:Re: system_schema.tables id and table uuid on
disk mismatch
It sounds like you have done some concurrent table creation/deletion
in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients),
which resulted in this mismatch. After you restarted the node,
Cassandra corrected it by discarding the old table ID and any data
associated with it. This is the expected behaviour. This issue has
already been fixed, and you can safely delete the data directory
with the incorrect table ID as it is no longer used by Cassandra.
You should now run a full repair on this node to ensure it has all
the data it owns. If you are /absolutely/ certain that the table
with different IDs have identical schema, and the gc_grace_seconds
hasn't past, you may move the data from the wrong data directory to
the correct data directory, and then restart the node or run
"nodetool refresh <keyspace> <table>" on the node before running the
full repair, this may save you some streaming time. However, if the
table schema is different, this may cause a havoc.
On 18/01/2024 05:21, ENES ATABERK wrote:
Hi all,
we have detected that table-uuid in linux file directory is
different from system_schema.tables id.
I have executed nodetool describe cluster and see only one schema
version in the cluster.
How we can fix this issue do anyone has any idea? Restarting the
nodes only create a new empty directory with
name system_schema.tables id directory but in this case i have two
directories old one has sstables with incorrect uuid new one has
correct uuid but empty.
thanks in advance
<https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
Bu elektronik posta ve onunla iletilen butun dosyalar sadece
gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel
kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz
bu elektronik postanin icerigini aciklamaniz, kopyalamaniz,
yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik
postayi derhal silmeniz gerekmektedir.
TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz
oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu
bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden,
alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki
gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini
yansitmayabilir
Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom
they are addressed. If you are not the intended recipient you are
hereby notified that any dissemination, forwarding, copying or use
of any of the information is strictly prohibited, and the e-mail
should immediately be deleted.
TURKCELL makes no warranty as to the accuracy or completeness of
any information contained in this message and hereby excludes any
liability of any kind for the information contained therein or for
the information transmission, reception, storage or use of such in
any way whatsoever. The opinions expressed in this message belong
to sender alone and may not necessarily reflect the opinions of
TURKCELL.
This e-mail has been scanned for all known computer viruses.