[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989834#comment-13989834 ] Jonathan Ellis commented on CASSANDRA-7144: --- You didn't give a version so I'm not 100% sure but I think you're seeing this assert: {code} int size = rm.modifications.size(); out.writeInt(size); assert size > 0; {code} which means that your client is sending a zero-length mutation to the server. do we not validate that in the native protocol [~thobbs]? > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989838#comment-13989838 ] Maxime Lamothe-Brassard commented on CASSANDRA-7144: I had mentioned it, probably in the wrong field since it got removed. 2.0.7, latest at the moment. Quick update, I ended up decomissioning the node, wiping /var/lib/cassandra/* and then re-joining the cluster (same install, just wiped the dir). It has 100% solved my problem (so far anyway, about 2 days of uptime). So it sounds like the data on disk was somehow corrupted somewhere between my client code update and repairs. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.jav
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992841#comment-13992841 ] Maxime Lamothe-Brassard commented on CASSANDRA-7144: I was not using prepared statements. I was doing INSERT and SELECT, no DELETE. I have some more info now. As I said I rebuilt the data on the node and it eliminated the problem. However, a day later, I found myself killing my ingestor (doing lots of INSERT), it's a python script using the new cassandra python-driver. When I did that I got the exception above. Thinking it was a "runtime" bug, I just kept going. Then the hinted-handoff started timing out on that box, so I restarted cassandra. From that point on, I would get the same exception without ever killing ingestors, at random interval. It seems as if killing the script during a query ended up sending data to cassandra that made it corrupt something on disk and that from that point on whenever it reached that part of the data on disk (I use that liberally, I just mean NOT directly from the script doing the ingestion) it would throw the exact same exception, leading to the timing out again. Restart of the cassandra node did nothing, I had to rebuild the data again. So now I'm very paranoid about killing my ingestion. The ingestion uses UNLOGGED BATCH for some of the data for performance as well as normal INSERT. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004910#comment-14004910 ] Jason Punyon commented on CASSANDRA-7144: - I'm Jason and I work at Stack Overflow. We're bit by this right now on our 2.0.6 cluster and we haven't wiped it out yet. We got to this state on Friday after a power outage took out our entire datacenter (https://twitter.com/JasonPunyon/status/467290879569719296). We have a 3-node cluster and we interact with it using the CqlSharp library. We do our reads and writes at QUORUM, some of them are batches. The first symptom of the problem was that both read and write operations were timing out. When we dug in, we found that nodes 1 and 3 were throwing errors (not enough responses), but node 2 would execute the exact same operations fine at quorum. Nodes 1 and 3 would execute reads fine at ONE. We're pretty interested in getting it fixed. Is there any way we can help? > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005085#comment-14005085 ] Tyler Hobbs commented on CASSANDRA-7144: Hi [~jason.punyon], thanks for your offer to help. The best way to help at the moment would be to help come up with a reproduceable test case, if you can. I have one other ticket on my queue in front of this one, so I should be able to give it proper attention pretty soon. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cas
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005303#comment-14005303 ] Jason Punyon commented on CASSANDRA-7144: - [~thobbs] We racked our brains a bit, we can't really conceive of a direction to go in to replicate it again (remember it's still live on our cluster, we can run any diagnostics you want), aside from rebuilding and then randomly bringing power down (what we did the first time). > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apach
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006517#comment-14006517 ] Tyler Hobbs commented on CASSANDRA-7144: [~jason.punyon] [~nesnub] can you provide any other details about the writes? I presume you were using prepared statements? How large were the batches? Roughly what do the individual inserts (within the batches or otherwise) look like? > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnab
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006847#comment-14006847 ] Maxime Lamothe-Brassard commented on CASSANDRA-7144: The writes were not using prepared statements. The batch contain average INSERT/UPDATEs, but I also have a counter update interleaved between the statements. Here is a sample: {code} self.queries.append( ( ( 'UPDATE met SET n = n + 1 WHERE id = %(id)s', { 'id' : k } ) ) self.queries.append( ( '''BEGIN UNLOGGED BATCH INSERT INTO om ( id, obj, otype ) VALUES ( %(id)s, %(obj)s, %(type)s ); UPDATE loc SET last = %(ts)s WHERE sid = %(sid)s AND otype = %(type)s AND id = %(id)s; INSERT INTO lid ( id, sid ) VALUES ( %(id)s, %(sid)s ); APPLY BATCH;''', { 'id' : k, 'obj' : unicode( obj[ 0 ] ).lower(), 'type' : obj[ 1 ], 'ts' : ts, 'sid' : ag } ) ) {code} > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.Thr
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016182#comment-14016182 ] sankalp kohli commented on CASSANDRA-7144: -- Another point here is that one of these assertion errors which is caused in at OutboundTcpConnection.java:151 will cause that thread to die. This will be very bad as it will not be recreated and it won't be able to talk to a node which had that connection. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016199#comment-14016199 ] sankalp kohli commented on CASSANDRA-7144: -- [~nesnub] [~jason.punyon] Are you using CAS based inserts? > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.j
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016381#comment-14016381 ] Richard Low commented on CASSANDRA-7144: We saw this in 3 separate clusters at roughly the same time. In 2 clusters it happened on one node only. On 1 cluster it happened on 3 nodes. Every time it happened, there were 3 errors in WRITE threads with stack trace: {code} java.lang.AssertionError at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) {code} and the IPs in the thread names are all adjacent in the ring (we have replication 3+3 in 2 DCs). 2-3 seconds following that, there are then stack traces in the MutationStage like: {code} java.lang.AssertionError at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) at org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:897) at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:870) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1961) {code} The number of these varies from 5 to 65. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffMan
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016443#comment-14016443 ] Richard Low commented on CASSANDRA-7144: In fact, the number of MutationStage AssertionErrors always come in groups of 5 (within 1ms of each other). > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016525#comment-14016525 ] Richard Low commented on CASSANDRA-7144: I have a reproduction. Not sure if it was the cause of our issue but it repros an identical situation. I have a 3 node cluster running on my box. I ran these cql commands: {code} cqlsh> create keyspace ks with replication = {'class':'SimpleStrategy', 'replication_factor':2}; cqlsh> use ks; cqlsh:ks> create table counter_test (a text primary key, b counter) with gc_grace_seconds = 0; cqlsh:ks> consistency all; Consistency level set to ALL. cqlsh:ks> update counter_test set b = b + 1 where a = 'a'; cqlsh:ks> delete from counter_test where a = 'a'; cqlsh:ks> update counter_test set b = b + 1 where a = 'a'; Request did not complete within rpc_timeout. {code} And in the logs: {code} ERROR [WRITE-/127.0.0.2] 2014-06-03 14:37:24,186 CassandraDaemon.java (line 196) Exception in thread Thread[WRITE-/127.0.0.2,5,main] java.lang.AssertionError at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) ERROR [MutationStage:156] 2014-06-03 14:37:26,301 CassandraDaemon.java (line 196) Exception in thread Thread[MutationStage:156,5,main] java.lang.AssertionError at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) at org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:897) at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:870) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1961) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} The problem is that the row tombstone has microsecond timestamps but the counter has milliseconds (I just created CASSANDRA-7346 for this). When the local counter mutation is applied, it is immediately removed because the row tombstone has a later timestamp. When we read it back, once the row tombstone becomes GCable (that's why I set gc_grace to 0) we get null for the row, so we don't add a mutation to the RowMutation object (in CounterMutation.makeReplicaionMutation()). When this mutation gets sent it kills the connection thread, times out so we write a hint, which causes the second assertion failure. We now have to wait until the row tombstone becomes GCed to be able to inc anything in this row again. Even if there wasn't CASSANDRA-7346, it is legitimate to insert a row tombstone with a later timestamp so this could still happen. We need to detect this case and handle it so the empty counter increment doesn't cause trouble. We can just say it's done because the counter is deleted and not bother to replicate. I'll keep looking in case there are other code paths that trigger this. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Tyler Hobbs > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraD
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016578#comment-14016578 ] Aleksey Yeschenko commented on CASSANDRA-7144: -- Attaching a simple 1.2 based patch. Only 1.2 and 2.0 are affected, the 2.1 implementation is not. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.u
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016586#comment-14016586 ] Jason Punyon commented on CASSANDRA-7144: - Having looked at our writes, the were almost exclusively batches (sized 100-1000) and a counter batch (max size 1000). We used CAS for exactly one table (our migrations table) which was used very infrequently (in our build, a few times a day). > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(St
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016588#comment-14016588 ] Richard Low commented on CASSANDRA-7144: [~jason.punyon] did you ever issue any row deletes to your counter table? [~iamaleksey] I don't see any patch > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.u
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016593#comment-14016593 ] Jason Punyon commented on CASSANDRA-7144: - [~rlow] Yes we did. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.co
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016596#comment-14016596 ] Richard Low commented on CASSANDRA-7144: [~jason.punyon] great, this is almost certainly the cause for you. You should check out CASSANDRA-7346 since row deletes on counter tables don't do at all what you expect. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProx
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016599#comment-14016599 ] Richard Low commented on CASSANDRA-7144: [~iamaleksey] I tried your patch (rebased for 2.0.6 where we're seeing the problem). It stops the AssertionError which is great, but I still get a timeout for the response. Do we need to poke the responseHandler too? > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > a
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016601#comment-14016601 ] Aleksey Yeschenko commented on CASSANDRA-7144: -- bq. Do we need to poke the responseHandler too? We do. Oh this is ugly-ish. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executo
[jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016792#comment-14016792 ] Richard Low commented on CASSANDRA-7144: Confirmed v2 works on 2.0.6. There is no AssertionError or timeout. I haven't verified it hasn't broken anything else though but it looks like it shouldn't. > CassandraDaemon RowMutation exception > - > > Key: CASSANDRA-7144 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7144 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / > 2 Cores in DigitalOcean. >Reporter: Maxime Lamothe-Brassard >Assignee: Aleksey Yeschenko > Fix For: 1.2.17, 2.0.9 > > Attachments: 7144-2.txt, 7144.txt > > > First time reporting a bug here, apologies if I'm not posting it in the right > space. > At what seems like random interval, on random nodes in random situations I > will get the following exception. After this the hinted handoff start timing > out and the node stops participating in the cluster. > I started seeing these after switching to the Cassandra Python-Driver from > the Python-CQL driver. > {noformat} > ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java > (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120) > at > org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251) > at > org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203) > at > org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151) > ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:118,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line > 198) Exception in thread Thread[MutationStage:117,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654) > at > org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137) > at > org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908) > at > org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881) > at > org.apache.cassandra.service.StorageProxy$Hi