[
https://issues.apache.org/jira/browse/CASSANDRA-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577146#comment-13577146
]
Yuki Morishita commented on CASSANDRA-5229:
-------------------------------------------
Looks like the problem is streaming session silently hangs if the exception
thrown is not IOException(like RuntimeException in CASSANDRA-5105). When the
error was IOException, then the node would send retry or session failure
message.
> after a IOException is thrown during streaming, streaming tasks hang in
> netstats
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-5229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5229
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.1
> Environment: Ubuntu 12.04
> Reporter: Michael Kjellman
> Priority: Critical
>
> After an IOExcpetion, streaming tasks marked as "successful" in the logs are
> hung in netstats
> With TRACE debugging on streaming on the receiving node everything about the
> sstable in the log (not very much)
> {code}
> INFO [AntiEntropyStage:1] 2013-02-07 11:23:44,717 StreamOut.java (line 151)
> Stream context metadata
> [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db
> sections=3068 progress=0/2785204713 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db
> sections=2696 progress=0/758409465 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db
> sections=3099 progress=0/238876436 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db
> sections=1166 progress=0/2125323 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db
> sections=2507 progress=0/515992757 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
> sections=3153 progress=0/994857654 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db
> sections=3116 progress=0/129398170 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db
> sections=217 progress=0/72286 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db
> sections=3146 progress=0/3357709019 - 0%], 27 sstables.
> INFO [AntiEntropyStage:1] 2013-02-07 11:23:52,964 StreamOut.java (line 151)
> Stream context metadata
> [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db
> sections=2930 progress=0/2799914560 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db
> sections=2590 progress=0/761266059 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db
> sections=2956 progress=0/241362497 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db
> sections=1153 progress=0/2125323 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db
> sections=2422 progress=0/522126371 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
> sections=3004 progress=0/998401202 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db
> sections=2974 progress=0/129722346 - 0%,
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db
> sections=220 progress=0/72286 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db
> sections=2998 progress=0/3375554099 - 0%], 27 sstables.
> {code}
> node that is streaming out thinks that the streaming session was successful
> {code}
> INFO [MiscStage:1] 2013-02-07 11:23:38,022 StreamOut.java (line 151) Stream
> context metadata
> [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db
> sections=1727 progress=0/210208515 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
> sections=1746 progress=0/119438030 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db
> sections=1681 progress=0/54498226 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db
> sections=16 progress=0/13490 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db
> sections=632 progress=0/70019542 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db
> sections=1644 progress=0/39870238 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db
> sections=1569 progress=0/208331077 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db
> sections=1572 progress=0/30870478 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db
> sections=167 progress=0/1845123 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db
> sections=1574 progress=0/287386471 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db
> sections=811 progress=0/103776521 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db
> sections=1539 progress=0/141864261 - 0%], 12 sstables.
> INFO [MiscStage:1] 2013-02-07 11:23:49,938 StreamOut.java (line 151) Stream
> context metadata
> [/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
> sections=3153 progress=0/994857654 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db
> sections=2507 progress=0/515992757 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db
> sections=3153 progress=0/131969347 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db
> sections=3068 progress=0/2784807967 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db
> sections=2696 progress=0/758409465 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db
> sections=3150 progress=0/1792868996 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db
> sections=3123 progress=0/240094510 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-929-Data.db
> sections=18 progress=0/29468 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db
> sections=217 progress=0/13490 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db
> sections=3124 progress=0/130320291 - 0%,
> /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db
> sections=3055 progress=0/1749539598 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db
> sections=1608 progress=0/13357410 - 0%,
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db
> sections=3096 progress=0/1241397203 - 0%], 13 sstables.
> INFO [Streaming to /10.8.25.132:2] 2013-02-07 11:24:18,780
> StreamReplyVerbHandler.java (line 44) Successfully sent
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
> to /10.8.25.132
> {code}
> node that is being streamed to has nothing in the logs about this particular
> sstable
> {code}
> Streaming from: /10.8.30.13
> evidence:
> /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db
> sections=3153 progress=218225400/994857654 - 21%
> _______
> Streaming from: /10.138.12.10
> evidence:
> /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db
> sections=3153 progress=218225400/994857654 - 21%
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
