[ https://issues.apache.org/jira/browse/CASSANDRA-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010456#comment-14010456 ]
Joaquin Casares commented on CASSANDRA-7308: -------------------------------------------- Yes, I'm recreating the keyspace. I'll edit the description to include this. > Dropped keyspace causes issues later > ------------------------------------ > > Key: CASSANDRA-7308 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7308 > Project: Cassandra > Issue Type: Bug > Reporter: Joaquin Casares > Labels: datastax_qa > > I had a keyspace that was accepting writes and was able to query the keyspace > via cqlsh. I re-ran my test application that drops the keyspace if it exists > and wrote some more. This time, however, any select request via cqlsh breaks > with a `Request did not complete within rpc_timeout.`. > This was seen in the system.log: > {CODE} > ERROR [ReadStage:61096] 2014-05-27 17:21:39,995 CassandraDaemon.java (line > 198) Exception in thread Thread[ReadStage:61096,5,main] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.FileNotFoundException: > /var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-36-Data.db > (No such file or directory) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1927) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: > /var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-36-Data.db > (No such file or directory) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1362) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1147) > at > org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:69) > at > org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1579) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1699) > at > org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:137) > at > org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1381) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1923) > ... 3 more > Caused by: java.io.FileNotFoundException: > /var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-36-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241) > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55) > ... 12 more > INFO [CompactionExecutor:708] 2014-05-27 17:21:42,810 ColumnFamilyStore.java > (line 787) Enqueuing flush of Memtable-compactions_in_progress@46337516(0/0 > serialized/live bytes, 1 ops) > WARN [MemoryMeter:1] 2014-05-27 17:21:42,810 Memtable.java (line 470) > setting live ratio to maximum of 64.0 instead of Infinity > INFO [MemoryMeter:1] 2014-05-27 17:21:42,811 Memtable.java (line 481) > CFS(Keyspace='system', ColumnFamily='compactions_in_progress') liveRatio is > 64.0 (just-counted was 64.0). calculation took 0ms for 0 cells > INFO [FlushWriter:92] 2014-05-27 17:21:42,811 Memtable.java (line 363) > Writing Memtable-compactions_in_progress@46337516(0/0 serialized/live bytes, > 1 ops) > INFO [FlushWriter:92] 2014-05-27 17:21:42,817 Memtable.java (line 403) > Completed flushing > /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-51-Data.db > (42 bytes) for commitlog position ReplayPosition(segmentId=1400882649502, > position=5391795) > INFO [CompactionExecutor:708] 2014-05-27 17:21:42,819 CompactionTask.java > (line 275) Compacted 4 sstables to > [/var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-87,]. > 56,962,552 bytes to 56,961,531 (~99% of original) in 77,261ms = > 0.703107MB/s. 5 total partitions merged to 2. Partition merge counts were > {1:1, 4:1, } > INFO [CompactionExecutor:708] 2014-05-27 17:21:42,820 ColumnFamilyStore.java > (line 787) Enqueuing flush of > Memtable-compactions_in_progress@1777281623(179/3463 serialized/live bytes, 9 > ops) > INFO [MemoryMeter:1] 2014-05-27 17:21:42,820 Memtable.java (line 481) > CFS(Keyspace='system', ColumnFamily='compactions_in_progress') liveRatio is > 17.862604748603353 (just-counted was 16.374301675977655). calculation took > 0ms for 7 cells > INFO [FlushWriter:92] 2014-05-27 17:21:42,820 Memtable.java (line 363) > Writing Memtable-compactions_in_progress@1777281623(179/3197 serialized/live > bytes, 9 ops) > INFO [FlushWriter:92] 2014-05-27 17:21:42,829 Memtable.java (line 403) > Completed flushing > /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-52-Data.db > (168 bytes) for commitlog position ReplayPosition(segmentId=1400882649502, > position=5392718) > INFO [CompactionExecutor:706] 2014-05-27 17:21:42,829 CompactionTask.java > (line 115) Compacting > [SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-51-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-50-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-52-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-49-Data.db')] > INFO [CompactionExecutor:708] 2014-05-27 17:21:42,830 CompactionTask.java > (line 115) Compacting > [SSTableReader(path='/var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-71-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-81-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-76-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-87-Data.db')] > INFO [CompactionExecutor:708] 2014-05-27 17:21:42,892 > CompactionController.java (line 192) Compacting large row > weathercql/historical:SFJ (527238028 bytes) incrementally > INFO [CompactionExecutor:706] 2014-05-27 17:21:42,894 CompactionTask.java > (line 275) Compacted 4 sstables to > [/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-53,]. > 607 bytes to 321 (~52% of original) in 64ms = 0.004783MB/s. 8 total > partitions merged to 4. Partition merge counts were {1:4, 2:2, } > INFO [CompactionExecutor:710] 2014-05-27 17:21:57,469 > CompactionController.java (line 192) Compacting large row > weathercql/historical:UAK (205611614 bytes) incrementally > ERROR [ReadStage:61106] 2014-05-27 17:22:20,563 CassandraDaemon.java (line > 198) Exception in thread Thread[ReadStage:61106,5,main] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.FileNotFoundException: > /var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-36-Data.db > (No such file or directory) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1927) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: > /var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-36-Data.db > (No such file or directory) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1362) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1147) > at > org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:69) > at > org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1579) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1699) > at > org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:137) > at > org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1381) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1923) > ... 3 more > Caused by: java.io.FileNotFoundException: > /var/lib/cassandra/data/weathercql/historical/weathercql-historical-jb-36-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241) > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55) > ... 12 more > INFO [CompactionExecutor:707] 2014-05-27 17:22:20,661 > CompactionController.java (line 192) Compacting large row > weathercql/historical:MAG (477808828 bytes) incrementally > INFO [CompactionExecutor:710] 2014-05-27 17:22:34,154 > CompactionController.java (line 192) Compacting large row > weathercql/historical:MAG (537030028 bytes) incrementally > {CODE} -- This message was sent by Atlassian JIRA (v6.2#6252)