[jira] [Commented] (CASSANDRA-3112) Make repair fail when an unexpected error occurs
[ https://issues.apache.org/jira/browse/CASSANDRA-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668807#comment-13668807 ] Yuki Morishita commented on CASSANDRA-3112: --- I'm working on CASSANDRA-5426 and this can be dup of that. CASSANDRA-5426 is targeting for 2.0.0 release. > Make repair fail when an unexpected error occurs > > > Key: CASSANDRA-3112 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3112 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: repair > Fix For: 2.0 > > Attachments: 0003-Report-streaming-errors-back-to-repair-v4.patch, > 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch > > > CASSANDRA-2433 makes it so that nodetool repair will fail if a node > participating to repair dies before completing his part of the repair. This > handles most of the situation where repair was previously hanging, but repair > can still hang if an unexpected error occurs during either the merkle tree > creation (an on-disk corruption triggers an IOError say) or during streaming > (though I'm not sure what could make streaming failed outside of 'one of the > node died' (besides a bug)). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3112) Make repair fail when an unexpected error occurs
[ https://issues.apache.org/jira/browse/CASSANDRA-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668760#comment-13668760 ] Jonathan Ellis commented on CASSANDRA-3112: --- What is the scope of this ticket? Should it be wontfixed or moved to 2.1? > Make repair fail when an unexpected error occurs > > > Key: CASSANDRA-3112 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3112 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: repair > Fix For: 2.0 > > Attachments: 0003-Report-streaming-errors-back-to-repair-v4.patch, > 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch > > > CASSANDRA-2433 makes it so that nodetool repair will fail if a node > participating to repair dies before completing his part of the repair. This > handles most of the situation where repair was previously hanging, but repair > can still hang if an unexpected error occurs during either the merkle tree > creation (an on-disk corruption triggers an IOError say) or during streaming > (though I'm not sure what could make streaming failed outside of 'one of the > node died' (besides a bug)). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3112) Make repair fail when an unexpected error occurs
[
https://issues.apache.org/jira/browse/CASSANDRA-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598641#comment-13598641
]
Jason Wee commented on CASSANDRA-3112:
--
In StreamOutSession.java, the logger in the method convict(...) has two holder
but only 1 variable given... missing another variable to log it?
logger.error("StreamOutSession {} failed because {} died or was
restarted/removed", endpoint);
> Make repair fail when an unexpected error occurs
>
>
> Key: CASSANDRA-3112
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3112
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
>Priority: Minor
> Labels: repair
> Fix For: 2.0
>
> Attachments: 0003-Report-streaming-errors-back-to-repair-v4.patch,
> 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch
>
>
> CASSANDRA-2433 makes it so that nodetool repair will fail if a node
> participating to repair dies before completing his part of the repair. This
> handles most of the situation where repair was previously hanging, but repair
> can still hang if an unexpected error occurs during either the merkle tree
> creation (an on-disk corruption triggers an IOError say) or during streaming
> (though I'm not sure what could make streaming failed outside of 'one of the
> node died' (besides a bug)).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3112) Make repair fail when an unexpected error occurs
[ https://issues.apache.org/jira/browse/CASSANDRA-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176420#comment-13176420 ] Vijay commented on CASSANDRA-3112: -- "But do you know what is the reason for it making no progress? Because unless we know what can cause it, not sure what to fix?" it is usually is in the Streaming phase, i think adding a SoTimeout might fix it... but it is so random i couldn't reproduce in my tests but definitely seeing it in production. "How can we "lose" messages, aren't tcp supposed to avoid this?" Once you send the message the other node might get restarted (without validation or starting any thing) or the sockets can get reset, Actually i think when i posted this message it was because of CASSANDRA-3577. There isnt something like hints or a retry on the messages sent for the repairs. I understand this isnt the scope of this ticket, but i still think there should be a way to orchestrate repairs with a little complicated logic and i will try to do some parts of it in the other ticket. > Make repair fail when an unexpected error occurs > > > Key: CASSANDRA-3112 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3112 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: repair > Fix For: 1.1 > > Attachments: 0003-Report-streaming-errors-back-to-repair-v4.patch, > 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch > > > CASSANDRA-2433 makes it so that nodetool repair will fail if a node > participating to repair dies before completing his part of the repair. This > handles most of the situation where repair was previously hanging, but repair > can still hang if an unexpected error occurs during either the merkle tree > creation (an on-disk corruption triggers an IOError say) or during streaming > (though I'm not sure what could make streaming failed outside of 'one of the > node died' (besides a bug)). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3112) Make repair fail when an unexpected error occurs
[
https://issues.apache.org/jira/browse/CASSANDRA-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163669#comment-13163669
]
Sylvain Lebresne commented on CASSANDRA-3112:
-
bq. 1) Stream session or the stream doesn't have any progress (Read Timeout/rpc
timeout - Socket timeout might help)
But do you know what is the reason for it making no progress? Because unless we
know what can cause it, not sure what to fix?
{quote}
2) Validation compaction completed but the result tree is sent but not received.
3) Repair request is sent but the receiving node didn't receive it.
{quote}
How can we "lose" messages, aren't tcp supposed to avoid this?
4) When we have a big repair which runs for hours it will be better to retry
the failed part rather than full retry.
Streaming is supposed to have some part of built-in retry, though I'm not sure
there is situation where it is actually useful. But if we talking like having a
repair fail because a node die and continuing it once the node is back up, then
that would be nice, but I'm pretty sure this will be mightily complicated. In
particular and to name only one difficulty, whether this is for the validation
compaction or the streaming itself, we likely will have a hard time making sure
that sstables haven't been compacted between the initial try and the retry (or
we'll risk hanging on obsolete sstables forever). But in principle, that would
be nice. Clearly not in the scope of this ticket in any case.
> Make repair fail when an unexpected error occurs
>
>
> Key: CASSANDRA-3112
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3112
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
>Priority: Minor
> Labels: repair
> Fix For: 1.0.6
>
> Attachments: 0003-Report-streaming-errors-back-to-repair-v4.patch,
> 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch
>
>
> CASSANDRA-2433 makes it so that nodetool repair will fail if a node
> participating to repair dies before completing his part of the repair. This
> handles most of the situation where repair was previously hanging, but repair
> can still hang if an unexpected error occurs during either the merkle tree
> creation (an on-disk corruption triggers an IOError say) or during streaming
> (though I'm not sure what could make streaming failed outside of 'one of the
> node died' (besides a bug)).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3112) Make repair fail when an unexpected error occurs
[ https://issues.apache.org/jira/browse/CASSANDRA-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161327#comment-13161327 ] Vijay commented on CASSANDRA-3112: -- Hi Sylvain, I have seen the following issues in the Repairs specially in AWS Multi DC deployments... 1) Stream session or the stream doesn't have any progress (Read Timeout/rpc timeout - Socket timeout might help) 2) Validation compaction completed but the result tree is sent but not received? 3) Repair request is sent but the receiving node didn't receive it? 4) When we have a big repair which runs for hours it will be better to retry the failed part rather than full retry. Do you think it is worth to address this in a separate ticket? else i will close CASSANDRA-3487. > Make repair fail when an unexpected error occurs > > > Key: CASSANDRA-3112 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3112 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: repair > Fix For: 1.0.6 > > Attachments: 0003-Report-streaming-errors-back-to-repair-v4.patch, > 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch > > > CASSANDRA-2433 makes it so that nodetool repair will fail if a node > participating to repair dies before completing his part of the repair. This > handles most of the situation where repair was previously hanging, but repair > can still hang if an unexpected error occurs during either the merkle tree > creation (an on-disk corruption triggers an IOError say) or during streaming > (though I'm not sure what could make streaming failed outside of 'one of the > node died' (besides a bug)). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
