Re: unconfigured table logtabl
Be really cautious here, this can be deceptive There are races in some versions of cassandra that can leave you with different combinations of cfid The cfid is on disk for schema It’s in memory for schema It’s used for the table path on disk Those three have to match for things to work properly and various races can make them mismatch If you have one ID for the path and a different in the schema table on disk and you bounce, the database throws away the old directory and you start fresh with a new empty directory So resetlocalschema May “fix” this but it may fix it by throwing away one copy of data If only one host is out of sync you may want to pretend it died and replace it. Alternatively a ton of inspection can tell you which variation of mismatch you have and you can correct it properly but it’s more work than I’m prepared to type today Sorry for the unpleasant reality. > On Apr 5, 2020, at 3:38 PM, Erick Ramirez wrote: > > >> Another suggestion before resetlocalschema. Try rolling restart all the >> nodes in the cluster and see if it fix the problem. After the restart all >> the nodes will use the same schema for the table. > > That's a little bit heavy-handed. 🙂 Resetting a node's schema is a simple, > online operation that doesn't involve a cluster-wide restart. Cheers!
Re: unconfigured table logtabl
> > Another suggestion before resetlocalschema. Try rolling restart all the > nodes in the cluster and see if it fix the problem. After the restart all > the nodes will use the same schema for the table. > That's a little bit heavy-handed. 🙂 Resetting a node's schema is a simple, online operation that doesn't involve a cluster-wide restart. Cheers!
Re: unconfigured table logtabl
Another suggestion before resetlocalschema. Try rolling restart all the nodes in the cluster and see if it fix the problem. After the restart all the nodes will use the same schema for the table. On Sunday, April 5, 2020, David Ni wrote: > Hi Erick > Thank you very much for your friendly note. > ERROR [AntiEntropyStage:1] 2020-04-04 13:57:09,614 > RepairMessageVerbHandler.java:177 - Table with id 21a3fa90-74c7-11ea-978a- > b556b0c3a5ea was dropped during prepare phase of repair > cassandra@cqlsh:system_schema> select keyspace_name,table_name,id from > tables where keyspace_name='oapi_dev' and table_name='logtabl'; > keyspace_name | table_name | id > ---++-- > oapi_dev |logtabl | 830028a0-7584-11ea-a277-bdf3d1289bdd > the table id does not match the id from system_schema.tables > how to fix it? > > > > > At 2020-04-04 14:44:16, "Erick Ramirez" > wrote: > > Is it possible someone else dropped then recreated the logtabl table? > Also, did you confirm that the missing table ID matches the ID of logtabl? > > On a friendly note, there are a number of users here like me who respond > to questions on the go. I personally find it difficult to read screenshots > on my phone so if it isn't too much trouble, it would be preferable if you > pasted the text here instead. Cheers! > > > > > > > > >
Re:Re:Re: Re: Re: unconfigured table logtabl
Hi Erick Thank you very much for your friendly note. ERROR [AntiEntropyStage:1] 2020-04-04 13:57:09,614 RepairMessageVerbHandler.java:177 - Table with id 21a3fa90-74c7-11ea-978a-b556b0c3a5ea was dropped during prepare phase of repair cassandra@cqlsh:system_schema> select keyspace_name,table_name,id from tables where keyspace_name='oapi_dev' and table_name='logtabl'; keyspace_name | table_name | id ---++-- oapi_dev |logtabl | 830028a0-7584-11ea-a277-bdf3d1289bdd the table id does not match the id from system_schema.tables how to fix it? At 2020-04-04 14:44:16, "Erick Ramirez" wrote: Is it possible someone else dropped then recreated the logtabl table? Also, did you confirm that the missing table ID matches the ID of logtabl? On a friendly note, there are a number of users here like me who respond to questions on the go. I personally find it difficult to read screenshots on my phone so if it isn't too much trouble, it would be preferable if you pasted the text here instead. Cheers!
Re: Re: Re: Re: unconfigured table logtabl
This is confirmation that you have a schema disagreement in your cluster: - 21a3fa90-74c7-11ea-978a-b556b0c3a5ea = Friday, April 3 05:07:44 PT - 830028a0-7584-11ea-a277-bdf3d1289bdd = Friday, April 3 01:24:18 PT The schema on the node where you ran that query has an older version of the table (created on Friday 1am PT) versus the expected table ID (created on Friday 5am PT). Try running a nodetool resetlocalschema to force the node to get the latest version from other nodes. Check the docs for details on the command if you need to. Cheers!
Re:Re: Re: Re: unconfigured table logtabl
Thank you very much for your friendly note. ERROR [AntiEntropyStage:1] 2020-04-04 13:57:09,614 RepairMessageVerbHandler.java:177 - Table with id 21a3fa90-74c7-11ea-978a-b556b0c3a5ea was dropped during prepare phase of repair cassandra@cqlsh:system_schema> select keyspace_name,table_name,id from tables where keyspace_name='oapi_dev' and table_name='logtabl'; keyspace_name | table_name | id ---++-- oapi_dev |logtabl | 830028a0-7584-11ea-a277-bdf3d1289bdd the table id does not match the id from system_schema.tables how to fix it? At 2020-04-04 14:44:16, "Erick Ramirez" wrote: Is it possible someone else dropped then recreated the logtabl table? Also, did you confirm that the missing table ID matches the ID of logtabl? On a friendly note, there are a number of users here like me who respond to questions on the go. I personally find it difficult to read screenshots on my phone so if it isn't too much trouble, it would be preferable if you pasted the text here instead. Cheers!
Re: Re: Re: unconfigured table logtabl
Is it possible someone else dropped then recreated the logtabl table? Also, did you confirm that the missing table ID matches the ID of logtabl? On a friendly note, there are a number of users here like me who respond to questions on the go. I personally find it difficult to read screenshots on my phone so if it isn't too much trouble, it would be preferable if you pasted the text here instead. Cheers!
Re:Re: Re: unconfigured table logtabl
Hi Erick here is the log from system.log but when I desc keyspace the table is still there At 2020-04-04 11:56:15, "Erick Ramirez" wrote: And from your review of the logs, what was the cause of the repair failures? Because that's likely to yield clues on the root cause. Cheers!
Re: Re: unconfigured table logtabl
And from your review of the logs, what was the cause of the repair failures? Because that's likely to yield clues on the root cause. Cheers!
Re:Re: unconfigured table logtabl
Thanks Erick I think there is no problem with schema version: At 2020-04-04 10:35:06, "Erick Ramirez" wrote: The symptoms in your post indicate that you have a schema disagreement in your cluster which you need to resolve. These 2 nodes in particular are the ones you need to investigate in the first place: [2020-04-03 20:07:44,242] Got negative replies from endpoints [10.17.140.69, 10.17.140.59] If you run nodetool describecluster on several nodes, it will likely indicate that the 2 nodes above are unreachable. You need to review the logs to determine the cause and the quickest workaround is to restart those 2 nodes. It is also the likely reason you were getting the "unconfigured table" error, i.e. the node doesn't know about that table. CF ID b979fcc0-75a3-11ea-a277-bdf3d1289bdd indicates the table was created earlier today (April 3 05:07:44 PT) and the table's schema didn't get propagated to all the nodes in the cluster hence the schema disagreement. Cheers! GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/.
Re: unconfigured table logtabl
The symptoms in your post indicate that you have a schema disagreement in your cluster which you need to resolve. These 2 nodes in particular are the ones you need to investigate in the first place: [2020-04-03 20:07:44,242] Got negative replies from endpoints > [10.17.140.69, 10.17.140.59] > If you run nodetool describecluster on several nodes, it will likely indicate that the 2 nodes above are unreachable. You need to review the logs to determine the cause and the quickest workaround is to restart those 2 nodes. It is also the likely reason you were getting the "unconfigured table" error, i.e. the node doesn't know about that table. CF ID b979fcc0-75a3-11ea-a277-bdf3d1289bdd indicates the table was created earlier today (April 3 05:07:44 PT) and the table's schema didn't get propagated to all the nodes in the cluster hence the schema disagreement. Cheers! GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/.
unconfigured table logtabl
Hello experts:
I am using cassandra 3.11.2 and having below problem:
cassandra@cqlsh> select * from oapi_dev.logtabl;
InvalidRequest: Error from server: code=2200 [Invalid query]
message="unconfigured table logtabl"
but when I descrbibe keyspace，it shows in the result:
cassandra@cqlsh> desc oapi_dev;
CREATE KEYSPACE oapi_dev WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE oapi_dev.logtabl (
id bigint,
start_time timestamp,
take_time text,
api_name text,
api_type text,
client_ip text,
create_time timestamp,
end_time timestamp,
exchange_url text,
origin text,
request_body text,
request_method text,
request_params text,
result text,
result_content text,
url text,
user_name text,
PRIMARY KEY (id, start_time, take_time)
) WITH CLUSTERING ORDER BY (start_time ASC, take_time ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
I also tried to scrub and repair but also failed:
./nodetool scrub oapi_dev logtabl
nodetool: Unknown keyspace/cf pair (oapi_dev.logtabl )
[cassandra@plat-ecloud01-db-cassandra01 bin]$ ./nodetool repair -full oapi_dev
[2020-04-03 20:07:44,018] Starting repair command #6
(b979fcc0-75a3-11ea-a277-bdf3d1289bdd), repairing keyspace oapi_dev with repair
options (parallelism: parallel, primary range: false, incremental: false, job
threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 768,
pull repair: false)
[2020-04-03 20:07:44,242] Got negative replies from endpoints [10.17.140.69,
10.17.140.59]
[2020-04-03 20:07:44,242] Repair command #6 finished with error
error: Repair job has failed with the error message: [2020-04-03 20:07:44,242]
Got negative replies from endpoints [10.17.140.69, 10.17.140.59]
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2020-04-03 20:07:44,242] Got negative replies from endpoints [10.17.140.69,
10.17.140.59]
please give me some suggestion thanks in advance!
