[ https://issues.apache.org/jira/browse/CASSANDRA-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903443#action_12903443 ]
Bob T. Terminal commented on CASSANDRA-1376: -------------------------------------------- can confirm, had this happen on a 6 node cluster on 3 nodes at about same time. Server 2 - ERROR [COMMIT-LOG-WRITER] 2010-08-27 09:51:30,942 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main] Server 4 - ERROR [COMMIT-LOG-WRITER] 2010-08-27 09:43:56,882 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main] Server 5 - ERROR [COMMIT-LOG-WRITER] 2010-08-27 09:43:56,891 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main] java.lang.AssertionError at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157) at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124) at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70) at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103) at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521) at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:619) > Assertion Exception in COMMIT-LOG-WRITER > ---------------------------------------- > > Key: CASSANDRA-1376 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1376 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.7 beta 1 > Environment: CentOS 5.2 > Trunc > Reporter: Arya Goudarzi > Assignee: Matthew F. Dennis > Priority: Critical > Fix For: 0.7 beta 2 > > > I have a 3 node cluster. The last version which I used to insert data was > Trunc on Friday August 6th at 2pm. On Monday the cluster was updated to > version Trunc on August 8th. When I tried to insert rows, all 3 nodes gave > this exception and the cluster became unresponsive to all requests causing > TSocket timeouts. The resolution was to restart all 3 nodes. > ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (line > 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157) > at > org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103) > at > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:636) > I am not sure how to reproduce this again but on Trunc from Monday August > 9th, one of my 3 nodes did not start giving the following exception and again > the resolution was to restart that node: > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:549) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174) > at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545) > ... 5 more > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError > at > org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408) > at > org.apache.cassandra.db.ColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 6 more > Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400) > ... 8 more > Caused by: java.lang.AssertionError > at > org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157) > at > org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70) > at > org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450) > at > org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75) > at > org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 1 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.