I upgraded to 0.7.2 from 0.7.0 which was upgraded from 0.6.8 and I gets the following Exception. I have 4 nodes cluster on 2 data centers (2 nodes on each). I see the error only on 2 nodes on the same data center. I didn't see this error on 0.7.0
ERROR [HintedHandoff:5] 2011-02-21 03:53:22,341 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:338) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:391) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:334) ... 6 more Caused by: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:416) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Caused by: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:51) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69) ... 19 more ERROR [HintedHandoff:5] 2011-02-21 03:53:22,354 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[HintedHandoff:5,1,main] java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:338) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:391) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:334) ... 6 more Caused by: java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:416) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Caused by: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:51) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69) ... 19 more ERROR [CompactionExecutor:1] 2011-02-21 03:53:22,353 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.io.IOError: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:416) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.EOFException: attempted to skip 1600940404 bytes but only skipped 52486 at org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:51) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69) ... 19 more Shimi