I am running 1.2.4 with Vnodes and have been writing at low volume. I have
doubled the volume and suddenly 3 of my 6 nodes are showing much higher load
than the others (30 vs 3) and tpstats show the mutation stage as completely
full (see below). I did find a FileNotFoundException that I pasted below which
appears to be caused by creating, dropping, and creating a keyspace (something
I did but 4 or 5 days ago). Anyone have any idea what's going on here?
Thanks
Keiths-MacBook-Pro:bin keith$ ./nodetool tpstats -h lxpcas005.nanigans.com
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 0 0 130990 0
0
RequestResponseStage 0 0 344216 0
0
MutationStage 128 452 3464036 0
0
ReadRepairStage 0 0 14131 0
0
ReplicateOnWriteStage 0 0 32872 0
0
GossipStage 1 611 6351 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage 0 0 9 0
0
MemtablePostFlusher 0 0 91 0
0
FlushWriter 0 0 60 0
27
MiscStage 0 0 0 0
0
commitlog_archiver 0 0 0 0
0
InternalResponseStage 0 0 3 0
0
HintedHandoff 1 1 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 54
BINARY 0
READ 0
MUTATION 8539
_TRACE 0
REQUEST_RESPONSE 0
ERROR [ReplicateOnWriteStage:95404] 2013-05-06 14:55:06,555
CassandraDaemon.java (line 174) Exception in thread
Thread[ReplicateOnWriteStage:95404,5,main]
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.FileNotFoundException:
/data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
/data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
at
org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at
org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
... 3 more
Caused by: java.io.FileNotFoundException:
/data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
... 21 more
