Error during compaction followed by error during commitlog replay
-----------------------------------------------------------------
Key: CASSANDRA-1365
URL: https://issues.apache.org/jira/browse/CASSANDRA-1365
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Environment: OSX 10.6.4
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
cassandra r982829
Reporter: Valentino Volonghi
Created a cassandra ring of 2 nodes. Node 1 seeds node 2 and node 2 seeds node
1.
1) Fresh nodes up and verify that they both saw each other.
2) Run: python stress.py -o insert -n 100000 -d 192.168.101.168 -y regular -c
20 -i 5
3) Run: ./bin/nodetool -h 192.168.101.168 flush Keyspace1
3) Run: ./bin/nodetool -h 192.168.101.168 snapshot
=> checking the console output of cassandra this is the logging I see:
{noformat}
INFO 18:27:06,597 switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281057918124.log',
position=95597256)
INFO 18:27:06,598 Enqueuing flush of memtable-locationi...@1029661208(52
bytes, 2 operations)
INFO 18:27:06,599 Writing memtable-locationi...@1029661208(52 bytes, 2
operations)
INFO 18:27:06,704 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db
DEBUG 18:27:06,705 key cache capacity for LocationInfo is 5
DEBUG 18:27:06,705 Checking to see if compaction of LocationInfo would be useful
INFO 18:27:06,706 Compacting
[org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db')]
DEBUG 18:27:06,706 Discarding 0
DEBUG 18:27:06,706 discard completed log segments for
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281057918124.log',
position=95597256), column family 0.
DEBUG 18:27:06,706 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db : 256
DEBUG 18:27:06,708 Not safe to delete commit log
CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281057874925.log);
dirty is 2, 3,
DEBUG 18:27:06,709 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db : 512
DEBUG 18:27:06,709 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db : 768
DEBUG 18:27:06,709 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db : 1024
DEBUG 18:27:06,709 Expected bloom filter size : 1024
DEBUG 18:27:06,709 Marking replay position 95597256 on commit log
CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281057918124.log)
ERROR 18:27:06,752 Uncaught exception in thread
Thread[CompactionExecutor:1,5,main]
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
DEBUG 18:27:06,846 Snapshot for system table data file
/var/lib/cassandra/data/system/LocationInfo-e-3-Filter.db created as
/var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-3-Filter.db
DEBUG 18:27:06,976 Snapshot for system table data file
/var/lib/cassandra/data/system/LocationInfo-e-1-Filter.db created as
/var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-1-Filter.db
DEBUG 18:27:07,091 Snapshot for system table data file
/var/lib/cassandra/data/system/LocationInfo-e-2-Filter.db created as
/var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-2-Filter.db
DEBUG 18:27:07,210 Snapshot for system table data file
/var/lib/cassandra/data/system/LocationInfo-e-4-Filter.db created as
/var/lib/cassandra/data/system/snapshots/1281058026597/LocationInfo-e-4-Filter.db
DEBUG 18:27:07,341 Snapshot for system table data file
/var/lib/cassandra/data/system/Migrations-e-1-Filter.db created as
/var/lib/cassandra/data/system/snapshots/1281058026597/Migrations-e-1-Filter.db
DEBUG 18:27:07,462 Snapshot for system table data file
/var/lib/cassandra/data/system/Schema-e-1-Filter.db created as
/var/lib/cassandra/data/system/snapshots/1281058026597/Schema-e-1-Filter.db
{noformat}
4) At this point simply stop and restart cassandra. The first restart would
present this logging, the subsequent ones similar logging:
{noformat}
INFO 18:32:08,747 Replaying
/var/lib/cassandra/commitlog/CommitLog-1281057874925.log,
/var/lib/cassandra/commitlog/CommitLog-1281057918124.log
DEBUG 18:32:08,749 Replaying
/var/lib/cassandra/commitlog/CommitLog-1281057874925.log starting at 8974
INFO 18:32:08,749 Finished reading
/var/lib/cassandra/commitlog/CommitLog-1281057874925.log
DEBUG 18:32:08,790 Replaying
/var/lib/cassandra/commitlog/CommitLog-1281057918124.log starting at 95589044
DEBUG 18:32:08,791 Reading mutation at 95589044
DEBUG 18:32:08,800 replaying mutation for system...@60c0c8b5:
{ColumnFamily(Statistics [SuperColumn(ColumnCount
[0000000000000001:false:8...@1281058015446,0000000000000002:false:8...@1281058015446,0000000000000003:false:8...@1281058015446,0000000000000004:false:8...@1281058015446,0000000000000005:false:8...@1281058015446,0000000000000006:false:8...@1281058015446,0000000000000007:false:8...@1281058015446,0000000000000008:false:8...@1281058015446,000000000000000a:false:8...@1281058015447,000000000000000c:false:8...@1281058015447,000000000000000e:false:8...@1281058015447,0000000000000011:false:8...@1281058015447,0000000000000014:false:8...@1281058015447,0000000000000018:false:8...@1281058015447,000000000000001d:false:8...@1281058015447,0000000000000023:false:8...@1281058015447,000000000000002a:false:8...@1281058015447,0000000000000032:false:8...@1281058015447,000000000000003c:false:8...@1281058015447,0000000000000048:false:8...@1281058015447,0000000000000056:false:8...@1281058015447,0000000000000067:false:8...@1281058015447,000000000000007c:false:8...@1281058015447,0000000000000095:false:8...@1281058015447,00000000000000b3:false:8...@1281058015447,00000000000000d7:false:8...@1281058015447,0000000000000102:false:8...@1281058015447,0000000000000136:false:8...@1281058015447,0000000000000174:false:8...@1281058015447,00000000000001be:false:8...@1281058015447,0000000000000217:false:8...@1281058015447,0000000000000282:false:8...@1281058015447,0000000000000302:false:8...@1281058015447,000000000000039c:false:8...@1281058015447,0000000000000455:false:8...@1281058015447,0000000000000533:false:8...@1281058015447,000000000000063d:false:8...@1281058015447,000000000000077c:false:8...@1281058015447,00000000000008fb:false:8...@1281058015447,0000000000000ac7:false:8...@1281058015447,0000000000000cef:false:8...@1281058015447,0000000000000f85:false:8...@1281058015447,00000000000012a0:false:8...@1281058015447,000000000000165a:false:8...@1281058015447,0000000000001ad2:false:8...@1281058015447,000000000000202f:false:8...@1281058015447,000000000000269f:false:8...@1281058015447,0000000000002e58:false:8...@1281058015447,000000000000379d:false:8...@1281058015447,00000000000042bc:false:8...@1281058015447,0000000000005015:false:8...@1281058015447,0000000000006019:false:8...@1281058015447,0000000000007351:false:8...@1281058015447,0000000000008a61:false:8...@1281058015447,000000000000a60e:false:8...@1281058015447,000000000000c744:false:8...@1281058015447,000000000000ef1e:false:8...@1281058015447,0000000000011ef1:false:8...@1281058015447,0000000000015854:false:8...@1281058015447,0000000000019d32:false:8...@1281058015447,000000000001efd6:false:8...@1281058015447,0000000000025301:false:8...@1281058015447,000000000002ca01:false:8...@1281058015447,00000000000358ce:false:8...@1281058015447,000000000004042a:false:8...@1281058015447,000000000004d1cc:false:8...@1281058015447,000000000005c88e:false:8...@1281058015447,000000000006f0aa:false:8...@1281058015447,00000000000853ff:false:8...@1281058015447,000000000009fe65:false:8...@1281058015447,00000000000bfe13:false:8...@1281058015447,00000000000e6417:false:8...@1281058015447,00000000001144e8:false:8...@1281058015447,000000000014b916:false:8...@1281058015447,000000000018de1a:false:8...@1281058015447,00000000001dd752:false:8...@1281058015447,000000000023cf2f:false:8...@1281058015447,00000000002af89f:false:8...@1281058015447,00000000003390bf:false:8...@1281058015447,00000000003de0e5:false:8...@1281058015447,00000000004a4113:false:8...@1281058015447,0000000000591ae4:false:8...@1281058015447,00000000006aed12:false:8...@1281058015447,0000000000804faf:false:8...@1281058015447,000000000099f938:false:8...@1281058015447,0000000000b8c4aa:false:8...@1281058015448,0000000000ddb8cc:false:8...@1281058015448,00000000010a10f5:false:8...@1281058015448,00000000013f478c:false:8...@1281058015448,00000000017f22a8:false:8...@1281058015448,0000000001cbc330:false:8...@1281058015448,000000000227b706:false:8...@1281058015448,0000000002960ed4:false:8...@1281058015448,00000000031a7832:false:8...@1281058015448,0000000003b95d09:false:8...@1281058015448,000000000478093e:false:8...@1281058015448,00000000055cd7e4:false:8...@1281058015448,00000000066f6978:false:8...@1281058015448,0000000007b8e4f6:false:8...@1281058015448,00000000094445f4:false:8...@1281058015448,000000000b1eba58:false:8...@1281058015448,000000000d5812d0:false:8...@1281058015448,00000000100349c6:false:8...@1281058015448,0000000013372554:false:8...@1281058015448,00000000170ef998:false:8...@1281058015448,000000001bab91ea:false:8...@1281058015448,00000000213448b2:false:8...@1281058015448,0000000027d8573c:false:8...@1281058015448,000000002fd068ae:false:8...@1281058015448,0000000039607d9e:false:8...@1281058015448,0000000044da3057:false:8...@1281058015448,00000000529f6d35:false:8...@1281058015448,]),SuperColumn(RowSize
[0000000000000001:false:8...@1281058015445,0000000000000002:false:8...@1281058015445,0000000000000003:false:8...@1281058015445,0000000000000004:false:8...@1281058015445,0000000000000005:false:8...@1281058015445,0000000000000006:false:8...@1281058015445,0000000000000007:false:8...@1281058015445,0000000000000008:false:8...@1281058015445,000000000000000a:false:8...@1281058015445,000000000000000c:false:8...@1281058015445,000000000000000e:false:8...@1281058015445,0000000000000011:false:8...@1281058015445,0000000000000014:false:8...@1281058015445,0000000000000018:false:8...@1281058015445,000000000000001d:false:8...@1281058015445,0000000000000023:false:8...@1281058015445,000000000000002a:false:8...@1281058015445,0000000000000032:false:8...@1281058015445,000000000000003c:false:8...@1281058015445,0000000000000048:false:8...@1281058015445,0000000000000056:false:8...@1281058015445,0000000000000067:false:8...@1281058015445,000000000000007c:false:8...@1281058015445,0000000000000095:false:8...@1281058015445,00000000000000b3:false:8...@1281058015445,00000000000000d7:false:8...@1281058015445,0000000000000102:false:8...@1281058015445,0000000000000136:false:8...@1281058015445,0000000000000174:false:8...@1281058015445,00000000000001be:false:8...@1281058015445,0000000000000217:false:8...@1281058015445,0000000000000282:false:8...@1281058015445,0000000000000302:false:8...@1281058015445,000000000000039c:false:8...@1281058015445,0000000000000455:false:8...@1281058015445,0000000000000533:false:8...@1281058015445,000000000000063d:false:8...@1281058015445,000000000000077c:false:8...@1281058015445,00000000000008fb:false:8...@1281058015445,0000000000000ac7:false:8...@1281058015445,0000000000000cef:false:8...@1281058015445,0000000000000f85:false:8...@1281058015445,00000000000012a0:false:8...@1281058015446,000000000000165a:false:8...@1281058015446,0000000000001ad2:false:8...@1281058015446,000000000000202f:false:8...@1281058015446,000000000000269f:false:8...@1281058015446,0000000000002e58:false:8...@1281058015446,000000000000379d:false:8...@1281058015446,00000000000042bc:false:8...@1281058015446,0000000000005015:false:8...@1281058015446,0000000000006019:false:8...@1281058015446,0000000000007351:false:8...@1281058015446,0000000000008a61:false:8...@1281058015446,000000000000a60e:false:8...@1281058015446,000000000000c744:false:8...@1281058015446,000000000000ef1e:false:8...@1281058015446,0000000000011ef1:false:8...@1281058015446,0000000000015854:false:8...@1281058015446,0000000000019d32:false:8...@1281058015446,000000000001efd6:false:8...@1281058015446,0000000000025301:false:8...@1281058015446,000000000002ca01:false:8...@1281058015446,00000000000358ce:false:8...@1281058015446,000000000004042a:false:8...@1281058015446,000000000004d1cc:false:8...@1281058015446,000000000005c88e:false:8...@1281058015446,000000000006f0aa:false:8...@1281058015446,00000000000853ff:false:8...@1281058015446,000000000009fe65:false:8...@1281058015446,00000000000bfe13:false:8...@1281058015446,00000000000e6417:false:8...@1281058015446,00000000001144e8:false:8...@1281058015446,000000000014b916:false:8...@1281058015446,000000000018de1a:false:8...@1281058015446,00000000001dd752:false:8...@1281058015446,000000000023cf2f:false:8...@1281058015446,00000000002af89f:false:8...@1281058015446,00000000003390bf:false:8...@1281058015446,00000000003de0e5:false:8...@1281058015446,00000000004a4113:false:8...@1281058015446,0000000000591ae4:false:8...@1281058015446,00000000006aed12:false:8...@1281058015446,0000000000804faf:false:8...@1281058015446,000000000099f938:false:8...@1281058015446,0000000000b8c4aa:false:8...@1281058015446,0000000000ddb8cc:false:8...@1281058015446,00000000010a10f5:false:8...@1281058015446,00000000013f478c:false:8...@1281058015446,00000000017f22a8:false:8...@1281058015446,0000000001cbc330:false:8...@1281058015446,000000000227b706:false:8...@1281058015446,0000000002960ed4:false:8...@1281058015446,00000000031a7832:false:8...@1281058015446,0000000003b95d09:false:8...@1281058015446,000000000478093e:false:8...@1281058015446,00000000055cd7e4:false:8...@1281058015446,00000000066f6978:false:8...@1281058015446,0000000007b8e4f6:false:8...@1281058015446,00000000094445f4:false:8...@1281058015446,000000000b1eba58:false:8...@1281058015446,000000000d5812d0:false:8...@1281058015446,00000000100349c6:false:8...@1281058015446,0000000013372554:false:8...@1281058015446,00000000170ef998:false:8...@1281058015446,000000001bab91ea:false:8...@1281058015446,00000000213448b2:false:8...@1281058015446,0000000027d8573c:false:8...@1281058015446,000000002fd068ae:false:8...@1281058015446,0000000039607d9e:false:8...@1281058015446,0000000044da3057:false:8...@1281058015446,00000000529f6d35:false:8...@1281058015446,000000006325b640:false:8...@1281058015446,0000000076fa0de6:false:8...@1281058015446,000000008ec5aa47:false:8...@1281058015446,00000000ab539922:false:8...@1281058015446,00000000cd97848f:false:8...@1281058015446,00000000f6b5d245:false:8...@1281058015446,00000001280d62b9:false:8...@1281058015446,0000000163434344:false:8...@1281058015446,00000001aa50b71e:false:8...@1281058015446,00000001ff940ef1:false:8...@1281058015446,0000000265e4debb:false:8...@1281058015446,00000002e0ac3e7a:false:8...@1281058015446,000000037401e492:false:8...@1281058015446,0000000424cf1249:false:8...@1281058015446,00000004f8f87c58:false:8...@1281058015446,00000005f7909536:false:8...@1281058015446,000000072913e641:false:8...@1281058015446,0000000897b17ab4:false:8...@1281058015446,]),])}
INFO 18:32:08,817 Finished reading
/var/lib/cassandra/commitlog/CommitLog-1281057918124.log
DEBUG 18:32:08,818 Finished waiting on mutations from recovery
INFO 18:32:08,819 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1281058328819.log
INFO 18:32:08,847 switching in a fresh Memtable for Statistics at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
position=0)
INFO 18:32:08,856 Enqueuing flush of memtable-statist...@713462035(7502 bytes,
2 operations)
INFO 18:32:08,857 Writing memtable-statist...@713462035(7502 bytes, 2
operations)
INFO 18:32:08,961 Completed flushing
/var/lib/cassandra/data/system/Statistics-e-2-Data.db
DEBUG 18:32:08,962 key cache capacity for Statistics is 2
DEBUG 18:32:08,966 Checking to see if compaction of Statistics would be useful
DEBUG 18:32:08,966 Discarding 4
DEBUG 18:32:08,968 discard completed log segments for
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
position=0), column family 4.
DEBUG 18:32:08,968 Marking replay position 0 on commit log
CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281058328819.log)
INFO 18:32:08,969 Recovery complete
DEBUG 18:32:08,970 Deleting CommitLog-1281057874925.log
DEBUG 18:32:08,971 Deleting CommitLog-1281057918124.log
INFO 18:32:08,974 Log replay complete
DEBUG 18:32:08,980 Estimating compactions for Super1
DEBUG 18:32:08,980 Estimating compactions for Standard1
DEBUG 18:32:08,981 Estimating compactions for LocationInfo
DEBUG 18:32:08,982 Estimating compactions for HintsColumnFamily
DEBUG 18:32:08,982 Estimating compactions for Migrations
DEBUG 18:32:08,982 Estimating compactions for Schema
DEBUG 18:32:08,982 Estimating compactions for Statistics
INFO 18:32:08,983 Cassandra version: 0.7.0-SNAPSHOT
INFO 18:32:08,983 Thrift API version: 10.0.0
DEBUG 18:32:08,983 Checking to see if compaction of Super1 would be useful
DEBUG 18:32:08,983 Checking to see if compaction of Standard1 would be useful
INFO 18:32:08,984 Compacting
[org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-32-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-33-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-34-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-35-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-e-36-Data.db')]
INFO 18:32:08,984 Saved Token found: 43976851516388127359611406752167831045
DEBUG 18:32:08,984 index size for bloom filter calc for file :
/var/lib/cassandra/data/Keyspace1/Standard1-e-32-Data.db : 10624
INFO 18:32:08,985 Saved ClusterName found: Test Cluster
DEBUG 18:32:08,985 index size for bloom filter calc for file :
/var/lib/cassandra/data/Keyspace1/Standard1-e-33-Data.db : 11264
DEBUG 18:32:08,985 index size for bloom filter calc for file :
/var/lib/cassandra/data/Keyspace1/Standard1-e-34-Data.db : 27136
DEBUG 18:32:08,985 index size for bloom filter calc for file :
/var/lib/cassandra/data/Keyspace1/Standard1-e-35-Data.db : 43008
DEBUG 18:32:08,985 index size for bloom filter calc for file :
/var/lib/cassandra/data/Keyspace1/Standard1-e-36-Data.db : 57984
DEBUG 18:32:08,985 Expected bloom filter size : 57984
INFO 18:32:08,985 Saved partitioner not found. Using
org.apache.cassandra.dht.RandomPartitioner
INFO 18:32:08,988 switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
position=276)
INFO 18:32:08,988 Enqueuing flush of memtable-locationi...@122395252(95 bytes,
2 operations)
INFO 18:32:08,989 Writing memtable-locationi...@122395252(95 bytes, 2
operations)
ERROR 18:32:09,041 Uncaught exception in thread
Thread[CompactionExecutor:1,5,main]
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
DEBUG 18:32:09,044 Checking to see if compaction of LocationInfo would be useful
INFO 18:32:09,044 Compacting
[org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db')]
DEBUG 18:32:09,044 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db : 256
DEBUG 18:32:09,044 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db : 512
DEBUG 18:32:09,045 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db : 768
DEBUG 18:32:09,045 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db : 1024
DEBUG 18:32:09,045 Expected bloom filter size : 1024
ERROR 18:32:09,048 Uncaught exception in thread
Thread[CompactionExecutor:1,5,main]
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
DEBUG 18:32:09,051 Checking to see if compaction of HintsColumnFamily would be
useful
DEBUG 18:32:09,052 Checking to see if compaction of Migrations would be useful
DEBUG 18:32:09,052 Checking to see if compaction of Schema would be useful
DEBUG 18:32:09,052 Checking to see if compaction of Statistics would be useful
INFO 18:32:09,108 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-e-5-Data.db
DEBUG 18:32:09,109 key cache capacity for LocationInfo is 6
DEBUG 18:32:09,110 Checking to see if compaction of LocationInfo would be useful
DEBUG 18:32:09,110 Discarding 0
INFO 18:32:09,111 Compacting
[org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-e-5-Data.db')]
DEBUG 18:32:09,111 discard completed log segments for
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1281058328819.log',
position=276), column family 0.
DEBUG 18:32:09,111 Marking replay position 276 on commit log
CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1281058328819.log)
DEBUG 18:32:09,111 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-1-Data.db : 256
DEBUG 18:32:09,112 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-2-Data.db : 512
DEBUG 18:32:09,112 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-3-Data.db : 768
DEBUG 18:32:09,112 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-4-Data.db : 1024
DEBUG 18:32:09,112 index size for bloom filter calc for file :
/var/lib/cassandra/data/system/LocationInfo-e-5-Data.db : 1280
DEBUG 18:32:09,112 Expected bloom filter size : 1280
ERROR 18:32:09,123 Uncaught exception in thread
Thread[CompactionExecutor:1,5,main]
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.NoSuchMethodError:
org.apache.cassandra.config.DatabaseDescriptor.getInMemoryCompactionLimit()J
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:132)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:46)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:333)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:151)
at
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:132)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
INFO 18:32:09,167 Starting up server gossip
DEBUG 18:32:09,248 attempting to connect to /127.0.0.1
DEBUG 18:32:09,252 clearing cached endpoints
DEBUG 18:32:09,252 clearing cached endpoints
DEBUG 18:32:09,252 attempting to connect to /192.168.101.51
INFO 18:32:09,276 Binding thrift service to /192.168.101.168:9160
INFO 18:32:09,278 Using TFramedTransport with a max frame size of 15728640
bytes.
INFO 18:32:09,280 Listening for thrift clients...
DEBUG 18:32:09,280 Will try to load mx4j now, if it's in the classpath
DEBUG 18:32:09,322 attempting to connect to /192.168.101.51
INFO 18:32:09,375 Node /192.168.101.51 is now part of the cluster
DEBUG 18:32:09,375 Resetting pool for /192.168.101.51
DEBUG 18:32:09,377 Node /192.168.101.51 state normal, token
135629853287756932550538670294184981659
DEBUG 18:32:09,377 clearing cached endpoints
DEBUG 18:32:09,378 clearing cached endpoints
DEBUG 18:32:09,378 No bootstrapping or leaving nodes -> empty pending ranges
for Keyspace1
INFO 18:32:09,381 mx4j successfuly loaded
HttpAdaptor version 3.0.2 started on port 8081
DEBUG 18:32:09,803 attempting to connect to /192.168.101.51
INFO 18:32:09,810 Started hinted handoff for endpoint /192.168.101.51
INFO 18:32:09,810 InetAddress /192.168.101.51 is now UP
INFO 18:32:09,811 Finished hinted handoff of 0 rows to endpoint /192.168.101.51
DEBUG 18:32:10,175 GC for ParNew: 5 ms, 16210960 reclaimed leaving 193987168
used; max is 1207828480
DEBUG 18:32:11,229 attempting to connect to /127.0.0.1
DEBUG 18:32:11,244 Disseminating load info ...
{noformat}
5) After some restarts the error changes into:
{noformat}
INFO 18:32:40,951 Replaying
/var/lib/cassandra/commitlog/CommitLog-1281058340642.log
DEBUG 18:32:40,953 Replaying
/var/lib/cassandra/commitlog/CommitLog-1281058340642.log starting at 276
DEBUG 18:32:40,953 Reading mutation at 276
DEBUG 18:32:40,956 replaying mutation for system...@77fe4169:
{ColumnFamily(LocationInfo [B:false:1...@1281058340821,])}
DEBUG 18:32:40,965 Reading mutation at 424
INFO 18:32:40,966 Finished reading
/var/lib/cassandra/commitlog/CommitLog-1281058340642.log
ERROR 18:32:40,967 Exception encountered during startup.
org.apache.cassandra.db.marshal.MarshalException: invalid UTF8 bytes [-64, -88,
101, 51]
at org.apache.cassandra.db.marshal.UTF8Type.getString(UTF8Type.java:43)
at org.apache.cassandra.db.Column.getString(Column.java:247)
at
org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:85)
at org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:379)
at org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:279)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
Exception encountered during startup.
org.apache.cassandra.db.marshal.MarshalException: invalid UTF8 bytes [-64, -88,
101, 51]
at org.apache.cassandra.db.marshal.UTF8Type.getString(UTF8Type.java:43)
at org.apache.cassandra.db.Column.getString(Column.java:247)
at
org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:85)
at org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:379)
at org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:279)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
{noformat}
6) Then bring down cassandra on the side that is raising errors. Cleanup and
keep autobootstrap=false. Put datafiles of the snapshot in
/var/lib/cassandra/data/Keyspace1/, start cassandra, it gossips the schema and
picks up some of the files but doesn't replay the log apparently so no errors
are displayed. Then stop and restart and while replaying the log I see those
errors again.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
