Thanks Aaron, I was able to complete the repair by scrubbing the column family on all three replicas.
Cheers 2012/1/4 aaron morton <aa...@thelastpickle.com> > I was able to scrub the node the repair that failed was running on. Are > you saying the error could be displayed on that node but the bad data > coming from another node ? > > Yes. The error occurred the node was receiving a data stream from another, > you will need to clean the source of the data. You can either crawl through > the logs or scrub the entire cluster. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 4/01/2012, at 9:15 AM, Philippe wrote: > > I was able to scrub the node the repair that failed was running on. Are > you saying the error could be displayed on that node but the bad data > coming from another node ? > > Log inspection also showed many of these, they seem to happen around when > a stream transfer finishes. > ERROR [Thread-550876] 2012-01-03 16:35:31,922 > AbstractCassandraDaemon.java (line 139) Fatal exception in thread > Thread[Thread-550876,5,main] > java.lang.IllegalArgumentException > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:586) > at > org.apache.cassandra.streaming.IncomingStreamReader.readnwrite(IncomingStreamReader.java:110) > at > org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:85) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) > > Thanks > > 2012/1/2 aaron morton <aa...@thelastpickle.com> > >> I would try to nodetool scrub the data on the node that that sent the bad >> data in the stream. You may be able to work which node from the logs, or it >> may be easier to just scrub them all. >> >> Hope that helps. >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 31/12/2011, at 12:20 AM, Philippe wrote: >> >> Hello, >> Running a combination of 0.8.6 and 0.8.8 with RF=3, I am getting the >> following while repairing one node (all other nodes completed successfully). >> Can I just stop the instance, erase the SSTable and restart cleanup ? >> Thanks >> >> ERROR [Thread-402484] 2011-12-29 14:51:03,687 >> AbstractCassandraDaemon.java (line 139) Fatal exception in thread >> Thread[Thread-402484,5,main] >> java.lang.RuntimeException: java.util.concurrent.ExecutionException: >> java.io.IOError: java.io.IOException: Corrupt (negative) value length >> encountered >> at >> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) >> at >> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) >> Caused by: java.util.concurrent.ExecutionException: java.io.IOError: >> java.io.IOException: Corrupt (negative) value length encountered >> at >> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at >> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138) >> ... 3 more >> >> >> > >
