hello everybody,
This morning, Monday!!!, I was checking on Cassandra cluster and I noticed
that all data was missing. I noticed the following error on each node (9
nodes in the cluster):
*2021-03-01 09:05:52,984 WARN [MessagingService-Incoming-/x.x.x.x]
IncomingTcpConnection.java:103 run UnknownColumnFamilyException reading
from socket; closingorg.apache.cassandra.db.UnknownColumnFamilyException:
Couldn't find table for cfId cba90a70-5c46-11e9-9e36-f54fe3235e69. If a
table was just created, this is likely due to the schema not being fully
propagated. Please wait for schema agreement on table creation. at
org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1533)
at
org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:758)
at
org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:697)
at
org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)*
I tried to query the keyspace and got this:
node1# cqlsh
Connected to Cassandra Cluster at x.x.x.x:9042.
[cqlsh 5.0.1 | Cassandra 3.11.5.1 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> select * from mykeyspace.mytable where id = 123935;
*InvalidRequest: Error from server: code=2200 [Invalid query]
message="Keyspace * *mykeyspace does not exist"*
Investigating on each node I found that all the *SStables exist*, so I
think data is still there but the keyspace vanished, "magically".
Other facts I can tell you are:
- I have been getting Anticompaction errors from 2 nodes due to the fact
the disk was almost full.
- the cluster was online friday
- this morning, Monday, the whole cluster was offline and I noticed the
problem of "missing keyspace"
- During the weekend the cluster has been subject to inserts and deletes
- I have a 9 node (HDD) Cassandra 3.11 cluster.
I really need help on this, how can I restore the cluster?
Thank you very much
Marco
