[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169416#comment-14169416
]
T Jake Luciani commented on CASSANDRA-7821:
---
bq. Of course writing a new retry policy would be acceptable too, a
BackingOffRetryPolicy.java or something alike?
Yes this is how I think it should be done
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Attachments: CASSANDRA-7821-2.1.patch
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169444#comment-14169444
]
Russell Alexander Spitzer commented on CASSANDRA-7821:
--
Np, I'll work on that when i get some free time.
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Attachments: CASSANDRA-7821-2.1.patch
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14160697#comment-14160697
]
Russell Alexander Spitzer commented on CASSANDRA-7821:
--
We would have to write a new retry policy I think, the current ones retry
immediately which causes problems when we are testing under load. Sometimes a
node will become unresponsive temporarily and break the test but if we gave it
a bit of time it would come back online. If we retry as fast as possible then a
node which has stopped responding will stop the entire test.
Code i'm looking at for retry policies on the JD:
https://github.com/datastax/java-driver/blob/55b60db5767af26d4cf4618c495e0f760096c25c/driver-core/src/main/java/com/datastax/driver/core/policies/DefaultRetryPolicy.java
Of course writing a new retry policy would be acceptable too, a
BackingOffRetryPolicy.java or something alike?
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Attachments: CASSANDRA-7821-2.1.patch
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155157#comment-14155157
]
Jonathan Ellis commented on CASSANDRA-7821:
---
[~tjake] to review
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Attachments: CASSANDRA-7821-2.1.patch
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155997#comment-14155997
]
T Jake Luciani commented on CASSANDRA-7821:
---
Hmm aren't there options we can pass to the java driver?
http://www.datastax.com/documentation/developer/java-driver/2.1/common/drivers/reference/automaticFailover_r.html
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Attachments: CASSANDRA-7821-2.1.patch
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1451#comment-1451
]
Russell Alexander Spitzer commented on CASSANDRA-7821:
--
One more commit, forgot to implement serializable
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Attachments: CASSANDRA-7821-2.1.patch
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7821) Add Optional Backoff on Retry to Cassandra Stress
[
https://issues.apache.org/jira/browse/CASSANDRA-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109626#comment-14109626
]
Russell Alexander Spitzer commented on CASSANDRA-7821:
--
Attached a patch which adds to simlpe options to C* Stress
{code}
backoff_strategy = {CONSTANT,LINEAR,EXPONENTIAL}
CONSTANT : A constant amount of seconds based on backoff_seconds
LINEAR : An amount of time based on the retry_num * backoff_seconds
EXPONENTIAL: An amount of time based on backoff_seconds * 2 ^ retry_num
backoff_seconds = #
The number of seconds to be used as a coefficent in the above strategies
{code}
https://github.com/RussellSpitzer/cassandra/compare/RussellSpitzer:cassandra-2.1...CASSANDRA-7821
I also bumped up the timeout for threads up to 10 minutes but ideally we would
pass through the max expected amount of retry time.
[~benedict] As usual your feedback would be extremely welcome
Add Optional Backoff on Retry to Cassandra Stress
-
Key: CASSANDRA-7821
URL: https://issues.apache.org/jira/browse/CASSANDRA-7821
Project: Cassandra
Issue Type: Improvement
Reporter: Russell Alexander Spitzer
Assignee: Russell Alexander Spitzer
Currently when stress is running against a cluster which occasionally has
nodes marked as down, it will almost immediately stop. This occurs because
the retry loop can execute extremely quickly if each execution terminates
with a {{com.datastax.driver.core.exceptions.NoHostAvailableException}} or
{{com.datastax.driver.core.exceptions.UnavailableException}}.
In case of these exceptions is will most likely be unable to succeed if the
retries are performed as fast as possible. To get around this, we could add
an optional delay on retries giving the cluster time to recover rather than
terminating the stress run.
We could make this configurable, with options such as:
* Constant # Delays the same amount after each retry
* Linear # Backoff a set amount * the trial number
* Exponential # Backoff set amount * 2 ^ trial number
This may also require adjusting the thread is stuck check to make sure that
the max retry timeout will not cause the thread to be terminated early.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
