[
https://issues.apache.org/jira/browse/CASSANDRA-18430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712460#comment-17712460
]
David Capwell commented on CASSANDRA-18430:
-------------------------------------------
new patch looks to have no timeouts, but j11 had the following errors
{code}
org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions
were thrown during test
at
org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1076)
at
org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1062)
at
org.apache.cassandra.distributed.test.topology.DecommissionAvoidTimeouts.test(DecommissionAvoidTimeouts.java:147)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Suppressed: java.lang.AssertionError: Unknown keyspace
system_distributed
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:325)
at
org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:163)
at
org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)
at
org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:163)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:152)
at
org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)
{code}
{code}
java.lang.AssertionError: Expected endpoint /127.0.0.6:7012 to be the last
replica, but found /127.0.0.4:7012;
[Full(/127.0.0.5:7012,(288230376151711613,432345564227567483]),
Full(/127.0.0.6:7012,(288230376151711613,432345564227567483]),
Full(/127.0.0.7:7012,(288230376151711613,432345564227567483]),
Full(/127.0.0.3:7012,(288230376151711613,432345564227567483]),
Full(/127.0.0.1:7012,(288230376151711613,432345564227567483]),
Full(/127.0.0.4:7012,(288230376151711613,432345564227567483])]
at
org.apache.cassandra.distributed.test.topology.DecommissionAvoidTimeouts$BB.sortedByProximity(DecommissionAvoidTimeouts.java:192)
{code}
{code}
java.lang.AssertionError: Expected endpoint /127.0.0.6:7012 to be the last
replica, but found /127.0.0.4:7012;
[Full(/127.0.0.5:7012,(4899916394579099453,5044031582654955323]),
Full(/127.0.0.3:7012,(4899916394579099453,5044031582654955323]),
Full(/127.0.0.1:7012,(4899916394579099453,5044031582654955323]),
Full(/127.0.0.6:7012,(4899916394579099453,5044031582654955323]),
Full(/127.0.0.7:7012,(4899916394579099453,5044031582654955323]),
Full(/127.0.0.4:7012,(4899916394579099453,5044031582654955323])]
at
org.apache.cassandra.distributed.test.topology.DecommissionAvoidTimeouts$BB.sortedByProximity(DecommissionAvoidTimeouts.java:192)
{code}
{code}
Expected endpoint /127.0.0.6:7012 to be the last replica, but found
/127.0.0.7:7012;
[Full(/127.0.0.2:7012,(1729382256910270313,1873497444986126183]),
Full(/127.0.0.6:7012,(1729382256910270313,1873497444986126183]),
Full(/127.0.0.1:7012,(1729382256910270313,1873497444986126183]),
Full(/127.0.0.3:7012,(1729382256910270313,1873497444986126183]),
Full(/127.0.0.5:7012,(1729382256910270313,1873497444986126183]),
Full(/127.0.0.7:7012,(1729382256910270313,1873497444986126183])]
at
org.apache.cassandra.distributed.test.topology.DecommissionAvoidTimeouts$BB.sortedByProximity(DecommissionAvoidTimeouts.java:192)
{code}
Last 3 errors are from the test making sure that the sorting worked as
expected. Due to severity being set the decomming node should be last out of
the set of 6, but It was second, which may lead to timeouts.
There are 2 cases I can think happened
1) test doesn't wait for gossip agreement to know every instance knows about
severity
2) nodes compute weights using a scheduler, so may need to refresh before
running tests
Ill try to look into improving both cases today
> When decommissioning should set Severity to limit traffic
> ---------------------------------------------------------
>
> Key: CASSANDRA-18430
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18430
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Coordination
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Fix For: 5.x
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When we are decommissioning we first set LEAVING, then LEFT, then disable
> networking; timeouts start to follow at this last stage. LEFT nodes should
> not be seen as part of the ring, but that may not be seen before the network
> is disabled. To better mitigate timeouts we should set severity as part of
> decom during the LEAVING phase; by setting severity reads should deprioritize
> traffic to this node.
> Remote DC writes do not leverage proximity or severity and instead use random
> for its select, writes may still timeout even though we know the node is
> leaving, and severity is set… to work in this model we should update remote
> DC writes to deprioritize nodes with severity set
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
