[ https://issues.apache.org/jira/browse/CASSANDRA-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175324#comment-15175324 ]
Oliver Lockwood edited comment on CASSANDRA-11288 at 3/2/16 9:30 AM: --------------------------------------------------------------------- I'm going to bump the severity to Major since I've got fairly reliable reproduction of this issue and it's manifested in 4-node environments as well as simpler 2-node ones. See the log output below for a fuller example. What's interesting is that the "agreed" schema AFTER the DROP TABLE command has been issued is exactly the same version ({c264a58b-a3a4-39f8-84ae-c0960b7e8b4c}) as that at the end of the previous DDL change. So it looks like the nodes agree that nothing has changed yet, even though the EVENT DROPPED TABLE has come through. {noformat} 2016-03-02 09:11:13 DEBUG SchemaChanger:61 - Executing cql statement ALTER TABLE table_8 WITH compaction = { 'class' : 'LeveledCompactionStrategy' }; 2016-03-02 09:11:13 DEBUG Cluster:2256 - Received event EVENT UPDATED TABLE testkeyspace.table_8, scheduling delivery 2016-03-02 09:11:13 DEBUG Cluster:2199 - Refreshing schema for testkeyspace.table_8 (TABLE) 2016-03-02 09:11:13 DEBUG ControlConnection:676 - Checking for schema agreement: versions are [d138c607-135d-3081-99ec-350661791061, c264a58b-a3a4-39f8-84ae-c0960b7e8b4c] 2016-03-02 09:11:14 DEBUG ControlConnection:676 - Checking for schema agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c] 2016-03-02 09:11:15 DEBUG ControlConnection:294 - [Control connection] Refreshing schema for testkeyspace.table_8 (TABLE) 2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement DROP TABLE table_1; 2016-03-02 09:11:15 DEBUG Cluster:2256 - Received event EVENT DROPPED TABLE testkeyspace.table_1, scheduling delivery 2016-03-02 09:11:15 DEBUG ControlConnection:676 - Checking for schema agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c] 2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement CREATE TABLE table_1 (party_id text primary key, profile text) WITH compaction = { 'class' : 'LeveledCompactionStrategy' }; Exception in thread "main" com.datastax.driver.core.exceptions.AlreadyExistsException: Table testkeyspace.table_1 already exists at com.datastax.driver.core.exceptions.AlreadyExistsException.copy(AlreadyExistsException.java:111) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) {noformat} was (Author: oliver.lockwood): I'm going to bump the severity to Major since I've got fairly reliable reproduction of this issue and it's manifested in 4-node environments as well as simpler 2-node ones. See the log output below for a fuller example. What's interesting is that the "agreed" schema AFTER the DROP TABLE command has been issued is exactly the same version ({c264a58b-a3a4-39f8-84ae-c0960b7e8b4c}) as that at the end of the previous DDL change. So it looks like the nodes agree that nothing has changed yet, even though the EVENT DROPPED TABLE has come through. {quote} 2016-03-02 09:11:13 DEBUG SchemaChanger:61 - Executing cql statement ALTER TABLE table_8 WITH compaction = { 'class' : 'LeveledCompactionStrategy' }; 2016-03-02 09:11:13 DEBUG Cluster:2256 - Received event EVENT UPDATED TABLE testkeyspace.table_8, scheduling delivery 2016-03-02 09:11:13 DEBUG Cluster:2199 - Refreshing schema for testkeyspace.table_8 (TABLE) 2016-03-02 09:11:13 DEBUG ControlConnection:676 - Checking for schema agreement: versions are [d138c607-135d-3081-99ec-350661791061, c264a58b-a3a4-39f8-84ae-c0960b7e8b4c] 2016-03-02 09:11:14 DEBUG ControlConnection:676 - Checking for schema agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c] 2016-03-02 09:11:15 DEBUG ControlConnection:294 - [Control connection] Refreshing schema for testkeyspace.table_8 (TABLE) 2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement DROP TABLE table_1; 2016-03-02 09:11:15 DEBUG Cluster:2256 - Received event EVENT DROPPED TABLE testkeyspace.table_1, scheduling delivery 2016-03-02 09:11:15 DEBUG ControlConnection:676 - Checking for schema agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c] 2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement CREATE TABLE table_1 (party_id text primary key, profile text) WITH compaction = { 'class' : 'LeveledCompactionStrategy' }; Exception in thread "main" com.datastax.driver.core.exceptions.AlreadyExistsException: Table testkeyspace.table_1 already exists at com.datastax.driver.core.exceptions.AlreadyExistsException.copy(AlreadyExistsException.java:111) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52){quote} > Schema agreement appears to be false positive following a DROP TABLE command > ---------------------------------------------------------------------------- > > Key: CASSANDRA-11288 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11288 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.14.439 (DSE 4.6.7) > 2 nodes OR 4 nodes > Connecting with Datastax Java driver 2.1.8 > Reporter: Oliver Lockwood > Priority: Minor > > As part of a schema migration operation, our application is calling the > following operations on the Java driver consecutively: > {quote} > session.execute("DROP TABLE table_name"); > session.execute("CREATE TABLE table_name (...)"); > {quote} > The second of these sometimes fails with a {{DriverException}} whose message > is "Table keyspace.table_name already exists". > In the schema migration operation, there's 4 of these drop/create pairings > and, although it's random which exact one fails, we've never managed to get > further than the third operation in approximately 10 attempts - so there's a > reasonably high proportion of failure. > I don't believe this is a driver issue because the driver is checking for > schema agreement (as per > https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/ControlConnection.java#L701) > and we are seeing a log message to that effect. > {quote} > c.d.d.c.ControlConnection - [] [] [] [] [] [] [] [] Checking for schema > agreement: versions are [02bce936-fddd-3bef-bb54-124d31bede57] > {quote} > This log message appears in between our own logs which say "Executing > statement DROP TABLE..." and "Executing statement CREATE TABLE...", so we can > be reasonably sure this log message refers to the DROP operation being viewed > as "in agreement". > Could this be a bug in the Cassandra server erroneously reporting that the > schemas are in agreement across the 2 nodes when, in fact, they are not? -- This message was sent by Atlassian JIRA (v6.3.4#6332)