[
https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-7144:
-----------------------------------
Description:
First time reporting a bug here, apologies if I'm not posting it in the right
space.
At what seems like random interval, on random nodes in random situations I will
get the following exception. After this the hinted handoff start timing out and
the node stops participating in the cluster.
I started seeing these after switching to the Cassandra Python-Driver from the
Python-CQL driver.
{noformat}
ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java
(line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java (line
198) Exception in thread Thread[WRITE-/10.128.194.70,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
198) Exception in thread Thread[MutationStage:118,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at
org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
at
org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
at
org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
at
org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
198) Exception in thread Thread[MutationStage:117,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at
org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
at
org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
at
org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
at
org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
The service must be restarted for the node to come back online. Let me know any
additional configuration details needed.
was:
First time reporting a bug here, apologies if I'm not posting it in the right
space.
At what seems like random interval, on random nodes in random situations I will
get the following exception. After this the hinted handoff start timing out and
the node stops participating in the cluster.
I started seeing these after switching to the Cassandra Python-Driver from the
Python-CQL driver.
ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java
(line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java (line
198) Exception in thread Thread[WRITE-/10.128.194.70,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
198) Exception in thread Thread[MutationStage:118,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at
org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
at
org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
at
org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
at
org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
198) Exception in thread Thread[MutationStage:117,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
at
org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
at
org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
at
org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
at
org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
The service must be restarted for the node to come back online. Let me know any
additional configuration details needed.
> CassandraDaemon RowMutation exception
> -------------------------------------
>
> Key: CASSANDRA-7144
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7144
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB /
> 2 Cores in DigitalOcean.
> Reporter: Maxime Lamothe-Brassard
>
> First time reporting a bug here, apologies if I'm not posting it in the right
> space.
> At what seems like random interval, on random nodes in random situations I
> will get the following exception. After this the hinted handoff start timing
> out and the node stops participating in the cluster.
> I started seeing these after switching to the Cassandra Python-Driver from
> the Python-CQL driver.
> {noformat}
> ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java
> (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
> ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java
> (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
> ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
> 198) Exception in thread Thread[MutationStage:118,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at
> org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
> at
> org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
> at
> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
> at
> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
> at
> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
> 198) Exception in thread Thread[MutationStage:117,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at
> org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
> at
> org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
> at
> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
> at
> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
> at
> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> The service must be restarted for the node to come back online. Let me know
> any additional configuration details needed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
