Hey Jonathan, Thanks for the response. I applied the patch to 0.7.4 and things have started working again nicely. Looks like this fix is going in 0.7.5. Any idea when 0.7.5 will be released? Thanks again!
On Wed, Mar 23, 2011 at 9:56 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > This looks like a bug > (https://issues.apache.org/jira/browse/CASSANDRA-2376), but not one > that would cause a crash. Actual process death is only caused by (a) > running out of memory or (2) JVM bugs. > > On Wed, Mar 23, 2011 at 9:17 PM, Sanjeev Kulkarni <sanj...@locomatix.com> > wrote: > > Hey guys, > > I have a one node system(with replication factor of 1) running cassandra. > > The machine has two disks. One is used as the commitlog and the other as > > cassandra's data directory. The node just had gotten unresponsive and had > to > > be hard rebooted. > > After restart, cassandra started off fine. But when I run a process that > > reads from it, after a while cassandra crashes. The log is attached. > > This is a new 0.7.4 installation and not a upgrade. I have run nodetool > > scrub and clean which ran fine. fsck on disks is also fine. > > Any help would be appreciated. > > ERROR [ReadStage:5] 2011-03-23 22:14:47,779 AbstractCassandraDaemon.java > > (line 112) Fatal exception in thread Thread[ReadStage:5,5,main] > > java.lang.AssertionError > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:176) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > > at > > > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > > at > > > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > > at org.apache.cassandra.db.Table.getRow(Table.java:333) > > at > > > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > > at > > > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > > at > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > ERROR [ReadStage:2] 2011-03-23 22:14:47,781 AbstractCassandraDaemon.java > > (line 112) Fatal exception in thread Thread[ReadStage:2,5,main] > > java.lang.AssertionError > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:176) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > > at > > > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > > at > > > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > > at org.apache.cassandra.db.Table.getRow(Table.java:333) > > at > > > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > > at > > > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > > at > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > ERROR [ReadStage:28] 2011-03-23 22:14:47,779 AbstractCassandraDaemon.java > > (line 112) Fatal exception in thread Thread[ReadStage:28,5,main] > > java.lang.AssertionError > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:176) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130) > > at > > > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72) > > at > > > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58) > > at > > > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > > at > > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > > at org.apache.cassandra.db.Table.getRow(Table.java:333) > > at > > > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > > at > > > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > > at > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > and more such lines. > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
