[ https://issues.apache.org/jira/browse/CASSANDRA-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849912#comment-13849912 ]
Jonathan Ellis commented on CASSANDRA-6493: ------------------------------------------- >From chat, this does not reproduce when CASSANDRA-6488 is reverted. > Exceptions when a second Datacenter is Added > -------------------------------------------- > > Key: CASSANDRA-6493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6493 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu, EC2 M1.large > Reporter: Russell Alexander Spitzer > > On adding a second datacenter several exceptions were raised. > Test outline: > Start 25 Node DC1 > Keyspace Setup Replication 3 > Begin insert against DC1 Using Stress > While the inserts are occuring > Start up 25 Node DC2 > Alter Keyspace to include Replication in 2nd DC > Run rebuild on DC2 > Wait for stress to finish > Run repair on Cluster > ... Some other operations > At the point when the second datacenter is added several warnings go off > because nodetool status is not functioning, and a few moments later the start > operation reports a failure because a node has not successfully turned on. > The first start attempt yielded the following exception on a node in the > second DC. > {code} > CassandraDaemon.java (line 464) Exception encountered during startup > java.lang.AssertionError: -7560216458456714666 not found in > -9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ..., > 9218575851928340117, 9219681798686280387 > at > org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) > at > org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) > at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) > at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) > {code} > The test automatically tries to restart nodes if they fail during startup, > The second attempt for this node succeeded but a 'nodetool status' still > failed and a different node in the second DC logged the following and failed > to start up. > {code} > ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) > Exception encountered during startup > java.util.ConcurrentModificationException > at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115) > at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169) > at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382) > at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444) > at > org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) > at > org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) > at > org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) > at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) > ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 > CassandraDaemon.java (line 191) Exception in thread > Thread[StorageServiceShutdownHook,5,main] > java.lang.NullPointerException > at > org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358) > at > org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373) > at > org.apache.cassandra.service.StorageService.access$000(StorageService.java:89) > at > org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.lang.Thread.run(Thread.java:724) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)