[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073234#comment-13073234 ]
Wojciech Meler commented on CASSANDRA-2973: ------------------------------------------- BINGO :) I've run scrub in "rolling" way :) So I've done prescrub snapshots of 4 remaining nodes. All logs are collected. (Un)fortunately I'm going on vacation so I can give you all you need on 15'th October. Data set is quite big - about 140GB per node, so I'll need some instructions what to extract for you. Snapshot file list from 12th node: -rw-r--r-- 2 cassandra cassandra 49714745792 Jul 30 02:20 fdb-g-1715-Data.db -rw-r--r-- 2 cassandra cassandra 4815616 Jul 30 02:20 fdb-g-1715-Filter.db -rw-r--r-- 2 cassandra cassandra 61556583 Jul 30 02:20 fdb-g-1715-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:20 fdb-g-1715-Statistics.db -rw-r--r-- 2 cassandra cassandra 3450580822 Jul 30 02:22 fdb-g-1717-Data.db -rw-r--r-- 2 cassandra cassandra 904096 Jul 30 02:22 fdb-g-1717-Filter.db -rw-r--r-- 2 cassandra cassandra 18024560 Jul 30 02:22 fdb-g-1717-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:22 fdb-g-1717-Statistics.db -rw-r--r-- 2 cassandra cassandra 3213527243 Jul 30 02:25 fdb-g-1721-Data.db -rw-r--r-- 2 cassandra cassandra 904336 Jul 30 02:25 fdb-g-1721-Filter.db -rw-r--r-- 2 cassandra cassandra 18026968 Jul 30 02:25 fdb-g-1721-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:25 fdb-g-1721-Statistics.db -rw-r--r-- 2 cassandra cassandra 98385535 Jul 30 03:20 fdb-g-1723-Data.db -rw-r--r-- 2 cassandra cassandra 247456 Jul 30 03:20 fdb-g-1723-Filter.db -rw-r--r-- 2 cassandra cassandra 3130599 Jul 30 03:20 fdb-g-1723-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 03:20 fdb-g-1723-Statistics.db -rw-r--r-- 2 cassandra cassandra 106856768 Jul 30 09:41 fdb-g-1734-Data.db -rw-r--r-- 2 cassandra cassandra 195376 Jul 30 09:41 fdb-g-1734-Filter.db -rw-r--r-- 2 cassandra cassandra 2713645 Jul 30 09:41 fdb-g-1734-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 09:41 fdb-g-1734-Statistics.db -rw-r--r-- 2 cassandra cassandra 111863768 Jul 30 17:14 fdb-g-1745-Data.db -rw-r--r-- 2 cassandra cassandra 194416 Jul 30 17:14 fdb-g-1745-Filter.db -rw-r--r-- 2 cassandra cassandra 2761017 Jul 30 17:14 fdb-g-1745-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 17:14 fdb-g-1745-Statistics.db -rw-r--r-- 2 cassandra cassandra 845886467 Jul 30 20:16 fdb-g-1750-Data.db -rw-r--r-- 2 cassandra cassandra 902896 Jul 30 20:16 fdb-g-1750-Filter.db -rw-r--r-- 2 cassandra cassandra 10419730 Jul 30 20:16 fdb-g-1750-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:16 fdb-g-1750-Statistics.db -rw-r--r-- 2 cassandra cassandra 40226865 Jul 30 21:12 fdb-g-1751-Data.db -rw-r--r-- 2 cassandra cassandra 66608 Jul 30 21:12 fdb-g-1751-Filter.db -rw-r--r-- 2 cassandra cassandra 1319135 Jul 30 21:12 fdb-g-1751-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12 fdb-g-1751-Statistics.db -rw-r--r-- 2 cassandra cassandra 10904996806 Jul 30 02:41 mailcache-g-3708-Data.db -rw-r--r-- 2 cassandra cassandra 254656 Jul 30 02:41 mailcache-g-3708-Filter.db -rw-r--r-- 2 cassandra cassandra 5073034 Jul 30 02:41 mailcache-g-3708-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:41 mailcache-g-3708-Statistics.db -rw-r--r-- 2 cassandra cassandra 18281871008 Jul 30 02:55 mailcache-g-3712-Data.db -rw-r--r-- 2 cassandra cassandra 968176 Jul 30 02:55 mailcache-g-3712-Filter.db -rw-r--r-- 2 cassandra cassandra 11904976 Jul 30 02:55 mailcache-g-3712-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:55 mailcache-g-3712-Statistics.db -rw-r--r-- 2 cassandra cassandra 35390850 Jul 30 04:41 mailcache-g-3717-Data.db -rw-r--r-- 2 cassandra cassandra 640 Jul 30 04:41 mailcache-g-3717-Filter.db -rw-r--r-- 2 cassandra cassandra 12533 Jul 30 04:41 mailcache-g-3717-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 04:41 mailcache-g-3717-Statistics.db -rw-r--r-- 2 cassandra cassandra 9067927959 Jul 30 11:02 mailcache-g-3727-Data.db -rw-r--r-- 2 cassandra cassandra 296416 Jul 30 11:02 mailcache-g-3727-Filter.db -rw-r--r-- 2 cassandra cassandra 3578551 Jul 30 11:02 mailcache-g-3727-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 11:02 mailcache-g-3727-Statistics.db -rw-r--r-- 2 cassandra cassandra 67351540 Jul 30 16:08 mailcache-g-3738-Data.db -rw-r--r-- 2 cassandra cassandra 1512 Jul 30 16:08 mailcache-g-3738-Filter.db -rw-r--r-- 2 cassandra cassandra 29793 Jul 30 16:08 mailcache-g-3738-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:08 mailcache-g-3738-Statistics.db -rw-r--r-- 2 cassandra cassandra 57132425 Jul 30 16:14 mailcache-g-3739-Data.db -rw-r--r-- 2 cassandra cassandra 1224 Jul 30 16:14 mailcache-g-3739-Filter.db -rw-r--r-- 2 cassandra cassandra 23985 Jul 30 16:14 mailcache-g-3739-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:14 mailcache-g-3739-Statistics.db -rw-r--r-- 2 cassandra cassandra 3832970739 Jul 30 20:17 mailcache-g-3748-Data.db -rw-r--r-- 2 cassandra cassandra 107056 Jul 30 20:17 mailcache-g-3748-Filter.db -rw-r--r-- 2 cassandra cassandra 1660163 Jul 30 20:17 mailcache-g-3748-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:17 mailcache-g-3748-Statistics.db -rw-r--r-- 2 cassandra cassandra 314153777 Jul 30 20:49 mailcache-g-3749-Data.db -rw-r--r-- 2 cassandra cassandra 6808 Jul 30 20:49 mailcache-g-3749-Filter.db -rw-r--r-- 2 cassandra cassandra 135155 Jul 30 20:49 mailcache-g-3749-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:49 mailcache-g-3749-Statistics.db -rw-r--r-- 2 cassandra cassandra 211709838 Jul 30 21:12 mailcache-g-3750-Data.db -rw-r--r-- 2 cassandra cassandra 4664 Jul 30 21:12 mailcache-g-3750-Filter.db -rw-r--r-- 2 cassandra cassandra 92163 Jul 30 21:12 mailcache-g-3750-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12 mailcache-g-3750-Statistics.db -rw-r--r-- 2 cassandra cassandra 8404555561 Jul 30 01:34 mta_logs-g-1485-Data.db -rw-r--r-- 2 cassandra cassandra 2551456 Jul 30 01:34 mta_logs-g-1485-Filter.db -rw-r--r-- 2 cassandra cassandra 50978377 Jul 30 01:34 mta_logs-g-1485-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 01:34 mta_logs-g-1485-Statistics.db -rw-r--r-- 2 cassandra cassandra 3759973372 Jul 30 01:37 mta_logs-g-1486-Data.db -rw-r--r-- 2 cassandra cassandra 1611616 Jul 30 01:37 mta_logs-g-1486-Filter.db -rw-r--r-- 2 cassandra cassandra 32166666 Jul 30 01:37 mta_logs-g-1486-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 01:37 mta_logs-g-1486-Statistics.db -rw-r--r-- 2 cassandra cassandra 4179780390 Jul 30 01:41 mta_logs-g-1487-Data.db -rw-r--r-- 2 cassandra cassandra 1507936 Jul 30 01:41 mta_logs-g-1487-Filter.db -rw-r--r-- 2 cassandra cassandra 30052328 Jul 30 01:41 mta_logs-g-1487-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 01:41 mta_logs-g-1487-Statistics.db -rw-r--r-- 2 cassandra cassandra 138623813 Jul 30 15:41 mta_logs-g-1507-Data.db -rw-r--r-- 2 cassandra cassandra 429616 Jul 30 15:41 mta_logs-g-1507-Filter.db -rw-r--r-- 2 cassandra cassandra 5115547 Jul 30 15:41 mta_logs-g-1507-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 15:41 mta_logs-g-1507-Statistics.db -rw-r--r-- 2 cassandra cassandra 493377854 Jul 30 18:15 mta_logs-g-1513-Data.db -rw-r--r-- 2 cassandra cassandra 1190896 Jul 30 18:15 mta_logs-g-1513-Filter.db -rw-r--r-- 2 cassandra cassandra 13534807 Jul 30 18:15 mta_logs-g-1513-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 18:15 mta_logs-g-1513-Statistics.db -rw-r--r-- 2 cassandra cassandra 36751337 Jul 30 19:14 mta_logs-g-1514-Data.db -rw-r--r-- 2 cassandra cassandra 107152 Jul 30 19:14 mta_logs-g-1514-Filter.db -rw-r--r-- 2 cassandra cassandra 2130178 Jul 30 19:14 mta_logs-g-1514-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 19:14 mta_logs-g-1514-Statistics.db -rw-r--r-- 2 cassandra cassandra 32420585 Jul 30 20:14 mta_logs-g-1515-Data.db -rw-r--r-- 2 cassandra cassandra 97784 Jul 30 20:14 mta_logs-g-1515-Filter.db -rw-r--r-- 2 cassandra cassandra 1943186 Jul 30 20:14 mta_logs-g-1515-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:14 mta_logs-g-1515-Statistics.db -rw-r--r-- 2 cassandra cassandra 30340652 Jul 30 21:12 mta_logs-g-1516-Data.db -rw-r--r-- 2 cassandra cassandra 90888 Jul 30 21:12 mta_logs-g-1516-Filter.db -rw-r--r-- 2 cassandra cassandra 1809686 Jul 30 21:12 mta_logs-g-1516-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12 mta_logs-g-1516-Statistics.db -rw-r--r-- 2 cassandra cassandra 574037723 Jul 30 02:26 quota_counters-g-476-Data.db -rw-r--r-- 2 cassandra cassandra 3778336 Jul 30 02:26 quota_counters-g-476-Filter.db -rw-r--r-- 2 cassandra cassandra 74766796 Jul 30 02:26 quota_counters-g-476-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:26 quota_counters-g-476-Statistics.db -rw-r--r-- 2 cassandra cassandra 66750073 Jul 30 16:14 quota_counters-g-500-Data.db -rw-r--r-- 2 cassandra cassandra 382512 Jul 30 16:14 quota_counters-g-500-Filter.db -rw-r--r-- 2 cassandra cassandra 7587562 Jul 30 16:14 quota_counters-g-500-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:14 quota_counters-g-500-Statistics.db -rw-r--r-- 2 cassandra cassandra 65704662 Jul 30 16:15 quota_counters-g-503-Data.db -rw-r--r-- 2 cassandra cassandra 409456 Jul 30 16:15 quota_counters-g-503-Filter.db -rw-r--r-- 2 cassandra cassandra 7134471 Jul 30 16:15 quota_counters-g-503-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:15 quota_counters-g-503-Statistics.db -rw-r--r-- 2 cassandra cassandra 496890534 Jul 30 16:20 quota_counters-g-505-Data.db -rw-r--r-- 2 cassandra cassandra 7479616 Jul 30 16:20 quota_counters-g-505-Filter.db -rw-r--r-- 2 cassandra cassandra 53756417 Jul 30 16:20 quota_counters-g-505-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:20 quota_counters-g-505-Statistics.db -rw-r--r-- 2 cassandra cassandra 101189405 Jul 30 19:13 quota_counters-g-508-Data.db -rw-r--r-- 2 cassandra cassandra 683296 Jul 30 19:13 quota_counters-g-508-Filter.db -rw-r--r-- 2 cassandra cassandra 10281151 Jul 30 19:13 quota_counters-g-508-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 19:13 quota_counters-g-508-Statistics.db -rw-r--r-- 2 cassandra cassandra 36645335 Jul 30 20:49 quota_counters-g-509-Data.db -rw-r--r-- 2 cassandra cassandra 182984 Jul 30 20:49 quota_counters-g-509-Filter.db -rw-r--r-- 2 cassandra cassandra 3633104 Jul 30 20:49 quota_counters-g-509-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:49 quota_counters-g-509-Statistics.db -rw-r--r-- 2 cassandra cassandra 10071816 Jul 30 21:12 quota_counters-g-510-Data.db -rw-r--r-- 2 cassandra cassandra 50536 Jul 30 21:12 quota_counters-g-510-Filter.db -rw-r--r-- 2 cassandra cassandra 1004098 Jul 30 21:12 quota_counters-g-510-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12 quota_counters-g-510-Statistics.db -rw-r--r-- 2 cassandra cassandra 118 Jul 30 02:25 quota-g-6-Data.db -rw-r--r-- 2 cassandra cassandra 496 Jul 30 02:25 quota-g-6-Filter.db -rw-r--r-- 2 cassandra cassandra 33 Jul 30 02:25 quota-g-6-Index.db -rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:25 quota-g-6-Statistics.db error log sample from that node: java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 10648 but 25701 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 10648 but 25701 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:73) ... 14 more 2011-07-30 21:24:48:340 CEST ERROR [ReadStage:1993][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:1993,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 27299 but 28532 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 27299 but 28532 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:73) ... 14 more 2011-07-30 21:26:43:823 CEST ERROR [ReadStage:1972][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:1972,5,main] java.lang.AssertionError at org.apache.cassandra.io.util.MappedFileDataInput.skipBytes(MappedFileDataInput.java:145) at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:227) at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:50) at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:72) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-07-30 21:26:53:869 CEST ERROR [ReadStage:1978][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:1978,5,main] java.lang.AssertionError at org.apache.cassandra.io.util.MappedFileDataInput.skipBytes(MappedFileDataInput.java:145) at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:227) at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:50) at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:72) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) > fatal errrors after nodetool cleanup > ------------------------------------ > > Key: CASSANDRA-2973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 > Project: Cassandra > Issue Type: Bug > Affects Versions: 0.8.1 > Reporter: Wojciech Meler > > after adding nodes to cluster & running cleanup I get scaring exceptions in > log: > 2011-07-30 00:00:05:506 CEST ERROR > [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal > exception in thread Thread[ReadStage:2335,5,main] > java.io.IOError: java.io.IOException: mmap segment underflow; remaining is > 4394 but 60165 requested > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:80) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67) > at > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) > at org.apache.cassandra.db.Table.getRow(Table.java:385) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 > but 60165 requested > at > org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) > at > org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) > at > org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:73) > ... 14 more > exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira