[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174690#comment-16174690
]
Sylvain Lebresne commented on CASSANDRA-13043:
----------------------------------------------
To me, it sounds like the problem we have is that we move endpoints from
'pending' to 'natural' endpoints too quickly after bootstrap, before the node
is actually fully ready, and that this is what we should be fixing (basically,
{{getLiveNaturalEndpoints}} shouldn't return node that are note truly ready,
and not knowing the ring yet is not ready to me). Maybe counters are the only
case where this is a problem today, but hand-fixing it only there feels like
we'll run into other problems like that in the future. That being said, the
whole boostrap/gossip-settling code is not the part I', the most confortable
with so I don't know how easy fixing that is (or if it's even
possible/reasonable to do in 3.x).
Other than that, I agree with you concern. If we want to stick with the current
approach at least as far 3.x goes for simplicity, then maybe we can simply
advertise rpc-ready at the end of {{CassandraDaemon.start}} but in all case? In
fact, I'm not sure why we only set it for the native protocol, and that might
simply be an oversight. At the very least, thrift is literally referred to
"rpc" in the yaml, so not setting {{RPC_READY}} for it feels wrong. And going a
step further, if you have explicitly asked to not start any of the client
server, then it wouldn't be a lie to set {{RPC_READY}} at the end of
{{start()}} in the sense of "every RPC servers you asked us to start is ready".
Anyway, maybe someone rely on this being only set for the native protocol, I
genuinely have no clue.
> UnavailabeException caused by counter writes forwarded to leaders without
> complete cluster view
> -----------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination
> Environment: Debian
> Reporter: Catalin Alexandru Zamfir
> Assignee: Stefano Ortolani
> Priority: Minor
> Fix For: 3.0.15, 3.11.1
>
> Attachments: 13043-3.0.patch, patch.diff
>
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-164,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-117,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
