[ https://issues.apache.org/jira/browse/CASSANDRA-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Jirsa updated CASSANDRA-12857: ----------------------------------- Labels: (was: proposed-wontfix) > Upgrade procedure between 2.1.x and 3.0.x is broken > --------------------------------------------------- > > Key: CASSANDRA-12857 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12857 > Project: Cassandra > Issue Type: Bug > Reporter: Alexander Yasnogor > Priority: Critical > Attachments: cassandra.schema > > > It is not possible safely to do Cassandra in place upgrade from 2.1.14 to > 3.0.9. > Distribution: deb packages from datastax community repo. > The upgrade was performed according to procedure from this docu: > https://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgrdCassandraDetails.html > Potential reason: The upgrade procedure creates corrupted system_schema and > this keyspace get populated in the cluster and kills it. > We started with one datacenter which contains 19 nodes divided to two racks. > First rack was successfully upgraded and nodetool describecluster reported > two schema versions. One for upgraded nodes, another for non-upgraded nodes. > On starting new version on a first node from the second rack: > {code:java} > INFO [main] 2016-10-25 13:06:12,103 LegacySchemaMigrator.java:87 - Moving 11 > keyspaces from legacy schema tables to the new schema keyspace (system_schema) > INFO [main] 2016-10-25 13:06:12,104 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@7505e6ac > INFO [main] 2016-10-25 13:06:12,200 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@64414574 > INFO [main] 2016-10-25 13:06:12,204 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@3f2c5f45 > INFO [main] 2016-10-25 13:06:12,207 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@2bc2d64d > INFO [main] 2016-10-25 13:06:12,301 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@77343846 > INFO [main] 2016-10-25 13:06:12,305 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@19b0b931 > INFO [main] 2016-10-25 13:06:12,308 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@44bb0b35 > INFO [main] 2016-10-25 13:06:12,311 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@79f6cd51 > INFO [main] 2016-10-25 13:06:12,319 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@2fcd363b > INFO [main] 2016-10-25 13:06:12,356 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@609eead6 > INFO [main] 2016-10-25 13:06:12,358 LegacySchemaMigrator.java:148 - > Migrating keyspace > org.apache.cassandra.schema.LegacySchemaMigrator$Keyspace@7eb7f5d0 > INFO [main] 2016-10-25 13:06:13,958 LegacySchemaMigrator.java:97 - > Truncating legacy schema tables > INFO [main] 2016-10-25 13:06:26,474 LegacySchemaMigrator.java:103 - > Completed migration of legacy schema tables > INFO [main] 2016-10-25 13:06:26,474 StorageService.java:521 - Populating > token metadata from system tables > INFO [main] 2016-10-25 13:06:26,796 StorageService.java:528 - Token > metadata: Normal Tokens: [HUGE LIST of tokens] > INFO [main] 2016-10-25 13:06:29,066 ColumnFamilyStore.java:389 - > Initializing ... > INFO [main] 2016-10-25 13:06:29,066 ColumnFamilyStore.java:389 - > Initializing ... > INFO [main] 2016-10-25 13:06:45,894 AutoSavingCache.java:165 - Completed > loading (2 ms; 460 keys) KeyCache cache > INFO [main] 2016-10-25 13:06:46,982 StorageService.java:521 - Populating > token metadata from system tables > INFO [main] 2016-10-25 13:06:47,394 StorageService.java:528 - Token > metadata: Normal Tokens:[HUGE LIST of tokens] > INFO [main] 2016-10-25 13:06:47,420 LegacyHintsMigrator.java:88 - Migrating > legacy hints to new storage > INFO [main] 2016-10-25 13:06:47,420 LegacyHintsMigrator.java:91 - Forcing a > major compaction of system.hints table > INFO [main] 2016-10-25 13:06:50,587 LegacyHintsMigrator.java:95 - Writing > legacy hints to the new storage > INFO [main] 2016-10-25 13:06:53,927 LegacyHintsMigrator.java:99 - Truncating > system.hints table > .... > INFO [main] 2016-10-25 13:06:56,572 MigrationManager.java:342 - Create new > table: > org.apache.cassandra.config.CFMetaData@242e5306[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing > sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, > bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, > default_time_to_live=0, memtable_flush_period_in_ms=3600000, > min_index_interval=128, max_index_interval=2048, > speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' > : 'NONE'}, > compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, > options={min_threshold=4, max_threshold=32}}, > compression=org.apache.cassandra.schema.CompressionParams@3fa913a4, > extensions={}},comparator=comparator(),partitionColumns=[[] | [client command > coordinator duration request started_at > parameters]],partitionKeyColumns=[ColumnDefinition{name=session_id, > type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, > position=0}],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[ColumnDefinition{name=client, > type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, > position=-1}, ColumnDefinition{name=command, > type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, > ColumnDefinition{name=session_id, > type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, > position=0}, ColumnDefinition{name=coordinator, > type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, > position=-1}, ColumnDefinition{name=request, > type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, > ColumnDefinition{name=started_at, > type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, > position=-1}, ColumnDefinition{name=duration, > type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, position=-1}, > ColumnDefinition{name=parameters, > type=org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type), > kind=REGULAR, position=-1}],droppedColumns={},triggers=[],indexes=[]] > INFO [GossipStage:1] 2016-10-25 13:06:57,121 StorageService.java:1969 - Node > /10.41.100.31 state jump to NORMAL > INFO [GossipStage:1] 2016-10-25 13:06:57,127 TokenMetadata.java:479 - > Updating topology for /10.41.100.31 > INFO [GossipStage:1] 2016-10-25 13:06:57,127 TokenMetadata.java:479 - > Updating topology for /10.41.100.31 > INFO [HANDSHAKE-/10.11.100.19] 2016-10-25 13:06:57,128 > OutboundTcpConnection.java:515 - Handshaking version with /10.11.100.19 > ..... > INFO [main] 2016-10-25 13:07:02,773 MigrationManager.java:342 - Create new > table: …………… > INFO [main] 2016-10-25 13:07:04,136 MigrationManager.java:302 - Create new > Keyspace: KeyspaceMetadata > {code} > But then all upgraded nodes reported many times the same error > {code:java} > ERROR [InternalResponseStage:12] 2016-10-25 13:07:26,891 > MigrationTask.java:96 - Configuration exception merging remote schema > org.apache.cassandra.exceptions.ConfigurationException: Column family > comparators do not match or are not compatible (found > comparator(org.apache.cassandra.db.marshal.UTF8Type, org.apac...... > at > org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:787) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:740) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1346) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1302) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1252) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > [apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > [apache-cassandra-3.0.9.jar:3.0.9] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > {code} > nodetool describecluster reported 4 different schema versions > 1. All nodes on old version > 2. 7 migrated nodes from the first rack > 3. 2 migrated nodes from the first rack > 4. 1 node from the second rack > > Meanwhile the cluster was fully responsible for reads and writes. > Anyway the migration was stopped at this point and further investigations > showed that there are corrupted records in system_schema.tables, > system_schema.columns contained duplicated broken records with \x00 instead > of letters. > {code:java} > dc1_tenant_ssd | \x00\x00\x00\x00\x00\x00 | > 0.001 | {'keys': 'ALL', 'rows_per_partition': 'ALL'} | > UP (on SSD) | {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} | > {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} | 1 | > 0 | 0 | {} | {'compound'} | > 864000 | 0ae08450-80b9-11e6-8bf1-0df6cc57511a | 2048 | > 0 | 128 | 0 | > 99PERCENTILE > dc1_tenant_ssd | \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 | > 0.001 | {'keys': 'ALL', 'rows_per_partition': 'ALL'} | UP (old CF > format, on SSD) | {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} | > {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} | 1 | > 0 | 0 | {} | {'dense'} | > 864000 | 16c420b0-78fd-11e6-ae98-ff8f609f3a2d | 2048 | > 0 | 128 | 0 | > 99PERCENTILE > dc1_tenant_ssd | \x00\x00\x00\x00\x00\x00 | > 0.001 | {'keys': 'ALL', 'rows_per_partition': 'ALL'} | > UT (on SSD) | {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} | > {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} | 1 | > 0 | 0 | {} | {'dense'} | > 864000 | c38bce70-78fc-11e6-ae98-ff8f609f3a2d | 2048 | > 0 | 128 | 0 | > 99PERCENTILE > dc1_tenant_ssd | user_p | > 0.001 | {'keys': 'ALL', 'rows_per_partition': 'ALL'} | > UP (on SSD) | {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} | > {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} | 1 | > 0 | 0 | {} | {'compound'} | > 864000 | 0ae08450-80b9-11e6-8bf1-0df6cc57511a | 2048 | > 0 | 128 | 0 | > 99PERCENTILE > dc1_tenant_ssd | user_p_oldcf | > 0.001 | {'keys': 'ALL', 'rows_per_partition': 'ALL'} | UP (old CF > format, on SSD) | {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} | > {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} | 1 | > 0 | 0 | {} | {'dense'} | > 864000 | 16c420b0-78fd-11e6-ae98-ff8f609f3a2d | 2048 | > 0 | 128 | 0 | > 99PERCENTILE > dc1_tenant_ssd | user_t | > 0.001 | {'keys': 'ALL', 'rows_per_partition': 'ALL'} | > UT (on SSD) | {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} | > {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} | 1 | > 0 | 0 | {} | {'dense'} | > 864000 | c38bce70-78fc-11e6-ae98-ff8f609f3a2d | 2048 | > 0 | 128 | 0 | > 99PERCENTILE > {code} > it is clear that system_schema was corrupted on every node based on > sstabledump output. > The strange thing is that before upgrading the whole cluster, one single node > was upgraded one day before and system_schema was OK before to roll out the > upgrade on other nodes. It was particularly checked. > later the upgraded nodes refused to restart due to the duplicates in > system_schema.tables with an exception: > {code:java} > java.lang.IllegalStateException: One row required, 2 found > at > org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:84) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:938) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) > [apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) > [apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) > [apache-cassandra-3.0.9.jar:3.0.9] > {code} > I am quite confident that this is not a hardware problem, so far tried to > perform the upgrade twice with the same results. > Yes, this very unfortunate migration scenario didn't affect only one node and > brought the cluster to unusable state where there were no ways back. So far > decommission didn't work between different versions and scrub removed all > data from tables in system_schema. > We ended up by exporting the data and removing upgraded nodes from the > cluster with its data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org