[
https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950897#comment-13950897
]
Benedict commented on CASSANDRA-6948:
-------------------------------------
Thanks. Regrettably those logs are all still well after the problems began
occurring (they appear to all be from today!), which has caused the logs to get
completely spammed. Do you have any earlier logs, or have they been trashed by
log recycling?
> Native & ReplicateOnWriteStage AssertionError: Callback already exists
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-6948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Keith Wright
> Assignee: Benedict
> Attachments: Screen Shot 2014-03-28 at 11.27.56 AM.png, Screen Shot
> 2014-03-28 at 11.29.24 AM.png, cassandra.yaml, logs.tar.gz, system.log.1.gz,
> system.log.gz
>
>
> We are using the datastax driver and have seen the following exceptions
> appear periodically in our cluster in "groups". The client will gradually
> start sending less write requests to the nodes where this occurs the most
> often. This may be related to CASSANDRA-6476. Exception from logs:
> ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638
> ErrorMessage.java (line 222) Unexpected exception during request
> java.lang.AssertionError: Callback already exists for id -1665979622!
> (CallbackInfo(target=/10.106.160.84,
> callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea,
> serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b))
> at
> org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549)
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601)
> at
> org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984)
> at
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449)
> at
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505)
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358)
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131)
> at
> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638
> CassandraDaemon.java (line 196) Exception in thread
> Thread[ReplicateOnWriteStage:102766,5,main]
> java.lang.AssertionError: Callback already exists for id -1665979620!
> (CallbackInfo(target=/10.106.160.84,
> callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75,
> serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b))
> at
> org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549)
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601)
> at
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806)
> at
> org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
