[
https://issues.apache.org/jira/browse/CASSANDRA-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15045964#comment-15045964
]
Paulo Motta commented on CASSANDRA-10448:
-----------------------------------------
[~mroi] [~piotrwest] [~Zefir01] there's a chance this is caused by
CASSANDRA-10791. You'll probably need to re-scrub your data with 2.1.12 or
2.2.4, before running repair again.
It would be great if you could try that out and report back the results.
Otherwise, please apply the previous patch I suggested and attach debug.log for
further investigation.
> "Unknown type 0" Stream failure on Repair
> -----------------------------------------
>
> Key: CASSANDRA-10448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10448
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.2.2
> 5 Nodes in Google Compute Engine
> Java 1.8.0_60
> Reporter: Omri Iluz
> Assignee: Yuki Morishita
> Fix For: 2.2.x
>
> Attachments: apache-cassandra-2.2.4-SNAPSHOT.jar, casslogs.txt,
> receiversystem.log, sendersystem.log
>
>
> While running repair after upgrading to 2.2.2 I am getting many stream fail
> errors:
> {noformat}
> [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330
> for range (59694553044959221,86389982480621619] failed with error [repair
> #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti
> vities, (59694553044959221,86389982480621619]] Sync failed between
> /10.240.81.104 and /10.240.134.221 (progress: 4%)
> {noformat}
> Logs from both sides of the stream:
> Sides 1 -
> {noformat}
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Creating new streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10
> files(469491729 bytes)
> ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.81.104 is complete
> WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
> Side 2 -
> {noformat}
> INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86
> - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061
> StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Starting streaming to /10.240.134.221
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063
> StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Beginning stream session with /10.240.134.221
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13
> files(517391317 bytes)
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.134.221 is complete
> ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> ... 11 common frames omitted
> WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
