Hello,
I ran in to a problem today where Cassandra 1.2.18 exhausted its number of
permitted open file descriptors (65,535). This node has 256 tokens (vnodes)
and runs in a test environment with relatively little traffic/data.
As best I could tell, the majority of the file descriptors open were for a
single SSTable '.db' file. Looking in the error logs I found quite a few
exceptions that looked to have been identical:
ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line
191) Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to
seek to position 29049 in /mnt/var/lib/cassandra/data/path/to/file.db (1855
bytes) in read-only mode
Upon further investigation, it turns out this file became 'read-only' after
the Cassandra node was gracefully restarted last week. I'd imagine this is
a discussion for another email thread.
I fixed the issue by running:
nodetool scrub <Keyspace>
nodetool repair <Keyspace>
Attached to this email is one of the log entries/stacktrace for this
exception.
Before opening a JIRA ticket I thought I'd reach out to the list to see if
anyone has seen any similar behavior as well as do a bit of source-diving
to try and verify that the descriptor is actually leaking.
Cheers!
-Tim
ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line 191)
Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek
to position 29049 in
/mnt/var/lib/cassandra/data/IzanagiQueue/WorkQueue/IzanagiQueue-WorkQueue-ic-1-Data.db
(1855 bytes) in read-only mode
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1626)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: unable to seek to position 29049
in
/mnt/var/lib/cassandra/data/IzanagiQueue/WorkQueue/IzanagiQueue-WorkQueue-ic-1-Data.db
(1855 bytes) in read-only mode
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1048)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:130)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:91)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:44)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:104)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1130)
at org.apache.cassandra.db.Table.getRow(Table.java:348)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1070)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1622)
... 3 more
