[ https://issues.apache.org/jira/browse/CASSANDRA-11264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241559#comment-15241559 ]
Marcus Olsson commented on CASSANDRA-11264: ------------------------------------------- bq. After having a look at your original patch I saw that a failed task will be re-prioritized against other scheduled jobs/tasks with a high priority (given its last run time will not be updated), so that's already a retry mechanism in itself. While this is true, I believe that this part should probably be reworked a bit. If we have a scenario where one particular job will always fail, we will end up in a loop where that job would get retried constantly which leads to starvation of other jobs. One option is to keep it simple and only run it once (by removing the retry logic) and also add a flag for the job which is used to determine when the job is allowed to run again. Something like: {code} execute() { runTasks(); if (allTasksWasSuccessful()) { nextRun = -1 lastRunTime = now; } else { nextRun = now + defaultWaitTime; } } {code} Then that flag would be used to avoid prioritizing the failing job against the other jobs until the {{defaultWaitTime}} has elapsed. This flag could also work nicely with the rejection policies (assuming that they estimate the time until the job can actually be run), especially if we would be able to reject repairs on a specific table rather than all tables. WDYT? bq. Rather than cluttering the scheduled repair mechanism with retry logic, I think that it's better to add a retry option to (non-scheduled) repair job, and do more fine grained retry on individual steps such as validation and sync, since this will be more effective against transient failures rather than retrying the whole task and potentially losing work of non-failed tasks. Great idea! If e.g. a validation would fail on one node, would we clean up the resources on that node by CASSANDRA-11190 (specifically about cleaning up resources, so that we can safely retry it) or would we need a separate way of doing that? bq. We can of course log warns and gather statistics when a scheduled task fails, but I think we should add retry support to repair independently of this. WDYT? Sounds good! > Repair scheduling - Failure handling and retry > ---------------------------------------------- > > Key: CASSANDRA-11264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11264 > Project: Cassandra > Issue Type: Sub-task > Reporter: Marcus Olsson > Assignee: Marcus Olsson > Priority: Minor > > Make it possible for repairs to be run again if they fail and clean up the > associated resources (validations and streaming sessions) before retrying. > Log a warning for each re-attempt and an error if it can't complete in X > times. The number of retries before considering the repair a failure could be > configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)