[
https://issues.apache.org/jira/browse/CASSANDRA-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Alexander Spitzer updated CASSANDRA-6493:
-------------------------------------------------
Description:
On adding a second datacenter several exceptions were raised.
Test outline:
Start 25 Node DC1
Keyspace Setup Replication 3
Begin insert against DC1 Using Stress
While the inserts are occuring
Start up 25 Node DC2
Alter Keyspace to include Replication in 2nd DC
Run rebuild on DC2
Wait for stress to finish
Run repair on Cluster
... Some other operations
At the point when the second datacenter is added several warnings go off
because nodetool status is not functioning, and a few moments later the start
operation reports a failure because a node has not successfully turned on.
The first start attempt yielded the following exception on a node in the second
DC.
{code}
CassandraDaemon.java (line 464) Exception encountered during startup
java.lang.AssertionError: -7560216458456714666 not found in
-9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ...,
9218575851928340117, 9219681798686280387
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
{code}
The test automatically tries to restart nodes if they fail during startup, The
second attempt for this node succeeded but a 'nodetool status' still failed and
a different node in the second DC logged the following and failed to start up.
{code}
ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exception
encountered during startup
java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemon.java
(line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358)
at
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373)
at
org.apache.cassandra.service.StorageService.access$000(StorageService.java:89)
at
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:724)
{code}
was:
On adding a second datacenter several exceptions were raised.
Test outline:
Start 25 Node DC1
Keyspace Setup Replication 3
Begin insert against DC1 Using Stress
While the inserts are occuring
Start up 25 Node DC2
Alter Keyspace to include Replication in 2nd DC
Run rebuild on DC2
Wait for stress to finish
Run repair on Cluster
... Some other operations
At the point when the second datacenter is added several warnings go off
because nodetool status is not functioning, and a few moments later the start
operation reports a failure because a node has not successfully turned on.
The first start attempt yielded the following exception on a node in the second
DC.
{code}
CassandraDaemon.java (line 464) Exception encountered during startup
java.lang.AssertionError: -7560216458456714666 not found in
-9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ...,
9218575851928340117, 9219681798686280387
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
{code}
The test automatically tries to restart nodes if they fail during startup, The
second attempt for this node succeeded but a nodetool still failed and a
different node in the second DC logged the following and failed to start up.
{code}
ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exception
encountered during startup
java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
at
org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
at
org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemon.java
(line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358)
at
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373)
at
org.apache.cassandra.service.StorageService.access$000(StorageService.java:89)
at
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:724)
{code}
> Exceptions when a second Datacenter is Added
> --------------------------------------------
>
> Key: CASSANDRA-6493
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6493
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu, EC2 M1.large
> Reporter: Russell Alexander Spitzer
>
> On adding a second datacenter several exceptions were raised.
> Test outline:
> Start 25 Node DC1
> Keyspace Setup Replication 3
> Begin insert against DC1 Using Stress
> While the inserts are occuring
> Start up 25 Node DC2
> Alter Keyspace to include Replication in 2nd DC
> Run rebuild on DC2
> Wait for stress to finish
> Run repair on Cluster
> ... Some other operations
> At the point when the second datacenter is added several warnings go off
> because nodetool status is not functioning, and a few moments later the start
> operation reports a failure because a node has not successfully turned on.
> The first start attempt yielded the following exception on a node in the
> second DC.
> {code}
> CassandraDaemon.java (line 464) Exception encountered during startup
> java.lang.AssertionError: -7560216458456714666 not found in
> -9222060278673125462, -9220751250790085193, ..... ALL THE TOKENS ...,
> 9218575851928340117, 9219681798686280387
> at
> org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
> at
> org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
> at
> org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
> at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
> at
> org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
> at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
> at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
> {code}
> The test automatically tries to restart nodes if they fail during startup,
> The second attempt for this node succeeded but a 'nodetool status' still
> failed and a different node in the second DC logged the following and failed
> to start up.
> {code}
> ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464)
> Exception encountered during startup
> java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
> at
> org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadata.java:752)
> at
> org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMetadata.java:696)
> at
> org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMetadata.java:703)
> at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:187)
> at
> org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(RangeStreamer.java:147)
> at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:121)
> at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
> ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876
> CassandraDaemon.java (line 191) Exception in thread
> Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358)
> at
> org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373)
> at
> org.apache.cassandra.service.StorageService.access$000(StorageService.java:89)
> at
> org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Thread.java:724)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
