Limited joy I would say :) No long term damage at least.
I ended up deleting (moving to another disk) all the sstables which fixed the
problem. I ran in to even more problems during repair (detailed in another
recent email) but it seems to have worked regardless. Just to be safe, I am in
the process of starting a ‘manual repair’ (copying SSTables from other nodes
for this particular CF then restarting and running a cleanup + major
compaction).
Any thoughts on what the root cause of this problem could be? It is somewhat
worrying that a CF can randomly become corrupt bringing down the whole node.
Cassandras handling of a corrupt CF (regardless of how rare an occurrence) is
less than elegant.
Dan
From: Aaron Morton [mailto:aa...@thelastpickle.com]
Sent: January-25-11 16:03
To: user@cassandra.apache.org
Subject: Re: Errors During Compaction
Dan how did you go with this? More joy, less joy or a continuation of the
current level of joy?
Aaron
On 24/01/2011, at 9:38 AM, Dan Hendry dan.hendry.j...@gmail.com wrote:
I have run into a strange problem and was hoping for suggestions on how to fix
it (0.7.0). When compaction occurs on one node for what appears to be one
specific column family, the following error pops up the Cassandra log.
Compaction apparently fails and temp files don’t get cleaned up. After a while
and what seems to be multiple failed compactions on the CF, the node runs out
of disk space and crashes. Not sure if it is a related problem or a function of
this being a heavily used column family but after failing to compact,
compaction restarts on the same CF exacerbating the issue.
Problems with this specific node started earlier this weekend when it crashed
with and OOM error. This is quite surprising since my memtable thresholds and
GC settings have been tuned to run with quite a bit of overhead during normal
operation (max heap usage usually = 10 GB on a 12 GB heap, average usage of
6-8 GB). I could not find anything abnormal in the logs which would prompt an
OOM.
I will look things over tomorrow and try to provide a bit more information on
the problem but as a solution, I was going to wipe out all SSTables for this CF
on this node and then run a repair. Far from ideal, is this a reasonable
solution?
ERROR [CompactionExecutor:1] 2011-01-23 14:10:29,855
AbstractCassandraDaemon.java (line 91) Fatal exception in thread
Thread[CompactionExecutor:1,1,RMI Runtime]
java.io.IOError: java.io.EOFException: attempted to skip -1983579368 bytes but
only skipped 0
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:78)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:143)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:135)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: attempted to skip -1983579368 bytes but only
skipped 0
at
org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:52)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:69)
... 20 more
Dan Hendry
(403) 660-2297
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.872 / Virus