[ https://issues.apache.org/jira/browse/CASSANDRA-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575318#comment-13575318 ]
Rick Branson commented on CASSANDRA-5229: ----------------------------------------- Seeing this as well. > after a IOException is thrown during streaming, streaming tasks hang in > netstats > -------------------------------------------------------------------------------- > > Key: CASSANDRA-5229 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5229 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.2.1 > Environment: Ubuntu 12.04 > Reporter: Michael Kjellman > Priority: Critical > > After an IOExcpetion, streaming tasks marked as "successful" in the logs are > hung in netstats > With TRACE debugging on streaming on the receiving node everything about the > sstable in the log (not very much) > {code} > INFO [AntiEntropyStage:1] 2013-02-07 11:23:44,717 StreamOut.java (line 151) > Stream context metadata > [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db > sections=3068 progress=0/2785204713 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db > sections=2696 progress=0/758409465 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db > sections=3099 progress=0/238876436 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db > sections=1166 progress=0/2125323 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db > sections=2507 progress=0/515992757 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db > sections=3153 progress=0/994857654 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db > sections=3116 progress=0/129398170 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db > sections=217 progress=0/72286 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db > sections=3146 progress=0/3357709019 - 0%], 27 sstables. > INFO [AntiEntropyStage:1] 2013-02-07 11:23:52,964 StreamOut.java (line 151) > Stream context metadata > [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db > sections=2930 progress=0/2799914560 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db > sections=2590 progress=0/761266059 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db > sections=2956 progress=0/241362497 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db > sections=1153 progress=0/2125323 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db > sections=2422 progress=0/522126371 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db > sections=3004 progress=0/998401202 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db > sections=2974 progress=0/129722346 - 0%, > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db > sections=220 progress=0/72286 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db > sections=2998 progress=0/3375554099 - 0%], 27 sstables. > {code} > node that is streaming out thinks that the streaming session was successful > {code} > INFO [MiscStage:1] 2013-02-07 11:23:38,022 StreamOut.java (line 151) Stream > context metadata > [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db > sections=1727 progress=0/210208515 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db > sections=1746 progress=0/119438030 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db > sections=1681 progress=0/54498226 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db > sections=16 progress=0/13490 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db > sections=632 progress=0/70019542 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db > sections=1644 progress=0/39870238 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db > sections=1569 progress=0/208331077 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db > sections=1572 progress=0/30870478 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db > sections=167 progress=0/1845123 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db > sections=1574 progress=0/287386471 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db > sections=811 progress=0/103776521 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db > sections=1539 progress=0/141864261 - 0%], 12 sstables. > INFO [MiscStage:1] 2013-02-07 11:23:49,938 StreamOut.java (line 151) Stream > context metadata > [/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db > sections=3153 progress=0/994857654 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db > sections=2507 progress=0/515992757 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db > sections=3153 progress=0/131969347 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db > sections=3068 progress=0/2784807967 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db > sections=2696 progress=0/758409465 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db > sections=3150 progress=0/1792868996 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db > sections=3123 progress=0/240094510 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-929-Data.db > sections=18 progress=0/29468 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db > sections=217 progress=0/13490 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db > sections=3124 progress=0/130320291 - 0%, > /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db > sections=3055 progress=0/1749539598 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db > sections=1608 progress=0/13357410 - 0%, > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db > sections=3096 progress=0/1241397203 - 0%], 13 sstables. > INFO [Streaming to /10.8.25.132:2] 2013-02-07 11:24:18,780 > StreamReplyVerbHandler.java (line 44) Successfully sent > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db > to /10.8.25.132 > {code} > node that is being streamed to has nothing in the logs about this particular > sstable > {code} > Streaming from: /10.8.30.13 > evidence: > /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db > sections=3153 progress=218225400/994857654 - 21% > _______ > Streaming from: /10.138.12.10 > evidence: > /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db > sections=3153 progress=218225400/994857654 - 21% > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira