[ https://issues.apache.org/jira/browse/CASSANDRA-10303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740258#comment-14740258 ]
zhaoyan commented on CASSANDRA-10303: ------------------------------------- These happen after we set “sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10” if not set this,there is another error: ERROR [RMI TCP Connection(19)-xxxxxxx] 2015-09-01 00:02:36,714 StorageService.java:1058 - Error while rebuilding node org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.1.8.jar:2.1.8] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:412) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:617) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:472) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:256) ~[apache-cassandra-2.1.8.jar:2.1.8] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60] ERROR [STREAM-OUT-/xxxxxxxxxx] 2015-09-01 00:02:36,714 StreamSession.java:502 - [Stream #04cd8620-4fd3-11e5-8d26-33ebdae8f95a] Streaming error occurred java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_60] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_60] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_60] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_60] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_60] at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) ~[apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351) [apache-cassandra-2.1.8.jar:2.1.8] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331) [apache-cassandra-2.1.8.jar:2.1.8] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60] > stream always hang up when use rebuild > -------------------------------------- > > Key: CASSANDRA-10303 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10303 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: jdk1.7 > cassandra 2.1.8 > Reporter: zhaoyan > > we add another datacenter. > use nodetool rebuild DC1 > stream from some node of old datacenter always hang up with these exception: > ERROR [Thread-1472] 2015-09-10 19:24:53,091 CassandraDaemon.java:223 - > Exception in thread Thread[Thread-1472,5,RMI Runtime] > java.lang.RuntimeException: java.io.IOException: Connection timed out > at com.google.common.base.Throwables.propagate(Throwables.java:160) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) > ~[apache-cassandra-2.1.8.jar:2.1.8] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60] > Caused by: java.io.IOException: Connection timed out > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_60] > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > ~[na:1.7.0_60] > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_60] > at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_60] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > ~[na:1.7.0_60] > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) > ~[na:1.7.0_60] > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) > ~[na:1.7.0_60] > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > ~[na:1.7.0_60] > at > org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) > ~[apache-cassandra-2.1.8.jar:2.1.8] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.8.jar:2.1.8] > ... 1 common frames omitted > i must restart node to stop current rebuild, and rebuild agagin and again to > success.... -- This message was sent by Atlassian JIRA (v6.3.4#6332)