ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
----------------------------------------------------------------------------
Key: CASSANDRA-1248
URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.6.3
Environment: Cassandra 0.6.3
java version "1.6.0_0"
IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
Ubuntu 8.10
6 node cluster
RF==3
Reporter: David King
A node died by running out of disk space. After growing the volume, trying to
execute reads dies with:
{quote}
ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
java.lang.ArrayIndexOutOfBoundsException
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
at
org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
at
org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{quote}
The clients then die with TimeoutExceptions.
It might be better to return an exception to the client immediately instead of
letting them die, and it would be helpful if this exception were accompanied
with a message indicating *which* SStable is malformed.
This command does *not* turn up the failed table (nor does the same with
sstablekeys):
{quote}
r...@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x;
/usr/local/cassandra/bin/sstable2json $x > /dev/null; done
{quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
