It seems on my cluster there are a few unserializable Rows. I'm trying to run
a repair on the nodes, but it also seems that the replica nodes have unreadable
or unserializable rows. The problem is, I cannot determine if the repair is
still going on, or if was interrupted because of these errors. It is unclear
because nothing else related to the repair show up in the logs. It's been
about 5 hours and I also don't see anything happening when I perform a
"nodetool netstats" on the nodes. The nodetool repair command is still
blocking from the console.
On the node I'm trying to repair, I see this after launching a "repair":
...
INFO [manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc] 2011-04-08
11:41:55,520 AntiEntropyService.java (line 770) Waiting for repair requests:
[#<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-7
9caafd7d3cc, /10.46.108.102, (DFS,main)>, #<TreeRequest
manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc, /10.46.108.101,
(DFS,main)>, #<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc
, /10.46.108.100, (DFS,main)>, #<TreeRequest
manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc, /10.47.108.101, (DFS,main)>]
...
In the log of the node "10.46.108.102" where the repair tries to compare the
replica data, I see a couple of the below exceptions a few minutes later.
Are the exceptions bad enough to cause the repair to fail?
ERROR [CompactionExecutor:1] 2011-04-08 11:43:01,177 PrecompactedRow.java (line
82) Skipping row DecoratedKey(1782314446006375058060694305099335169,
4d657373616765456e726963686d656e743a31343236) in /va
r/lib/cassandra/data/DFS/main-f-177-Data.db
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
at
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [CompactionExecutor:1] 2011-04-08 11:43:53,762 PrecompactedRow.java (line
82) Skipping row DecoratedKey(8073554114801607394928746621229606383,
34393734663734382d316330302d346164372d613333372d3162
34303866613933333832) in /var/lib/cassandra/data/DFS/main-f-177-Data.db
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
:
nodetool netstats reports:
Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 526207
Responses n/a 0 1747991
