[ https://issues.apache.org/jira/browse/CASSANDRA-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620117#comment-13620117 ]
Brandon Williams commented on CASSANDRA-5418: --------------------------------------------- You have corruption and need to run scrub first. That said, we could probably at least abort the repair session in this case. What do you think [~yukim]? > repair freezing > --------------- > > Key: CASSANDRA-5418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5418 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.2.2 > Environment: 5 nodes, RackInferring snitch, OpenVZ VMs, Centos 6, > Oracle JVM with JNA enabled. > Reporter: Igor Ivanov > > When I run *nodetool repair* on cas01 node it get's stuck at some point. > I see following exceptions in cas01 system.log: > {quote} > ERROR [Streaming to /10.10.45.60:28] 2013-04-02 09:03:55,353 > CassandraDaemon.java (line 132) Exception in thread Thread[Streaming to > /10.10.45.60:28,5,main] > java.lang.RuntimeException: java.io.EOFException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(Unknown Source) > at > org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) > at > org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) > at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ... 3 more > ERROR [Thread-2076] 2013-04-02 09:07:12,261 CassandraDaemon.java (line 132) > Exception in thread Thread[Thread-2076,5,main] > java.lang.AssertionError: incorrect row data size 130921 written to > /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-3660-Data.db; > correct is 131074 > at > org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285) > at > org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238) > at > org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78) > {quote} > On other machines there are some exceptions too: > {quote} > ERROR [Thread-1424] 2013-04-02 09:07:12,248 CassandraDaemon.java (line 132) > Exception in thread Thread[Thread-1424,5,main] > java.lang.AssertionError: incorrect row data size 130921 written to > /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-2268-Data.db; > correct is 131074 > at > org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285) > at > org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238) > at > org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78) > ERROR [Streaming to /10.10.45.58:55] 2013-04-02 09:07:12,263 > CassandraDaemon.java (line 132) Exception in thread Thread[Streaming to > /10.10.45.58:55,5,main] > java.lang.RuntimeException: java.io.EOFException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(Unknown Source) > at > org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193) > at > org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114) > at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ... 3 more > {quote} > Then I see frozen status in *nodetool netstats* and repair never completes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira