[ https://issues.apache.org/jira/browse/CASSANDRA-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Alexander Spitzer updated CASSANDRA-6493: ------------------------------------------------- Description: On adding a second datacenter several exceptions were raised. Test outline: Start 25 Node DC1 Keyspace Setup Replication 3 Begin insert against DC1 Using Stress While the inserts are occuring Start up 25 Node DC2 Alter Keyspace to include Replication in 2nd DC Run rebuild on DC2 Wait for stress to finish Run repair on Cluster ... Some other operations At the point when the second datacenter is added several warnings go off because nodetool status is not functioning, and a few moments later the start operation reports a failure because a node has not successfully turned on. The first start attempt yielded the following exception on a node in the second DC. {code} CassandraDaemon.java (line 464) Exception encountered during startup java.lang.AssertionError: -7560216458456714666 not found in -9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ..., 9218575851928340117, 9219681798686280387 at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) {code} The test automatically tries to restart nodes if they fail during startup, The second attempt for this node succeeded but a 'nodetool status' still failed and a different node in the second DC logged the following and failed to start up. {code} ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exception encountered during startup java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444) at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemon.java (line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:89) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:724) {code} was: On adding a second datacenter several exceptions were raised. Test outline: Start 25 Node DC1 Keyspace Setup Replication 3 Begin insert against DC1 Using Stress While the inserts are occuring Start up 25 Node DC2 Alter Keyspace to include Replication in 2nd DC Run rebuild on DC2 Wait for stress to finish Run repair on Cluster ... Some other operations At the point when the second datacenter is added several warnings go off because nodetool status is not functioning, and a few moments later the start operation reports a failure because a node has not successfully turned on. The first start attempt yielded the following exception on a node in the second DC. {code} CassandraDaemon.java (line 464) Exception encountered during startup java.lang.AssertionError: -7560216458456714666 not found in -9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ..., 9218575851928340117, 9219681798686280387 at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) {code} The test automatically tries to restart nodes if they fail during startup, The second attempt for this node succeeded but a nodetool still failed and a different node in the second DC logged the following and failed to start up. {code} ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exception encountered during startup java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444) at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemon.java (line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:89) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:724) {code} > Exceptions when a second Datacenter is Added > -------------------------------------------- > > Key: CASSANDRA-6493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6493 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu, EC2 M1.large > Reporter: Russell Alexander Spitzer > > On adding a second datacenter several exceptions were raised. > Test outline: > Start 25 Node DC1 > Keyspace Setup Replication 3 > Begin insert against DC1 Using Stress > While the inserts are occuring > Start up 25 Node DC2 > Alter Keyspace to include Replication in 2nd DC > Run rebuild on DC2 > Wait for stress to finish > Run repair on Cluster > ... Some other operations > At the point when the second datacenter is added several warnings go off > because nodetool status is not functioning, and a few moments later the start > operation reports a failure because a node has not successfully turned on. > The first start attempt yielded the following exception on a node in the > second DC. > {code} > CassandraDaemon.java (line 464) Exception encountered during startup > java.lang.AssertionError: -7560216458456714666 not found in > -9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ..., > 9218575851928340117, 9219681798686280387 > at > org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) > at > org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) > at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) > at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) > {code} > The test automatically tries to restart nodes if they fail during startup, > The second attempt for this node succeeded but a 'nodetool status' still > failed and a different node in the second DC logged the following and failed > to start up. > {code} > ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) > Exception encountered during startup > java.util.ConcurrentModificationException > at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115) > at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169) > at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382) > at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444) > at > org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696) > at > org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187) > at > org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147) > at > org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121) > at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) > ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 > CassandraDaemon.java (line 191) Exception in thread > Thread[StorageServiceShutdownHook,5,main] > java.lang.NullPointerException > at > org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358) > at > org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373) > at > org.apache.cassandra.service.StorageService.access$000(StorageService.java:89) > at > org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.lang.Thread.run(Thread.java:724) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)