GitHub user rick-kilgore opened a pull request:
https://github.com/apache/incubator-storm/pull/254
[STORM-495] KafkaSpout retries with exponential backoff
This is an implementation of the improvement request STORM-495.
The KafkaSpout keeps track of failed messages through multiple retries, and
delays their resubmission to the topology with an exponentially increasing
delay time. The spout does not terminate the retry cycle (it is my conviction
that it should not do so, because it cannot report context about the failure
that happened to abort the reqeust), it only handles delaying the retries. A
bolt in the topology is still expected to eventually call ack() instead of
fail() to stop the cycle.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HBOCodeLabs/incubator-storm retries
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/254.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #254
----
commit cac70e77d3417e13082495aaac126f25a6d07eff
Author: Rick Kilgore <[email protected]>
Date: 2014-08-23T04:49:08Z
exponential backoff for failed messages
commit f9d2948a9e2dadbbb99128fd6570038e2096eabd
Author: Rick Kilgore <[email protected]>
Date: 2014-09-03T22:04:32Z
Merge branch 'master' into retryDelay
commit 6138e945894c2e0b95666909af51fd336862ce6d
Author: Rick Kilgore <[email protected]>
Date: 2014-09-06T06:57:17Z
Merge branch 'master' into retryDelay
commit 1c748a7fde11ebc7d209c16ad2b30bf19a316395
Author: Rick Kilgore <[email protected]>
Date: 2014-09-06T07:08:12Z
Merge pull request #1 from rick-kilgore/retryDelay
exponential backoff for failed messages
moving repo to HBOCodeLabs
commit 04b64bc66d5c29ba77c455f0125200ed8335376f
Author: rick-kilgore <[email protected]>
Date: 2014-09-08T20:49:18Z
add comment and different default for retry
commit 05336764408921f3ebcc4ac5b3e82da10e713bea
Author: rick-kilgore <[email protected]>
Date: 2014-09-08T20:58:01Z
default settings = no retry delay
default settings = no retry delay - i.e., preserve legacy behavior
commit 79e3efe2558d252f2122facdb0ce5e69c9c62104
Author: Rick Kilgore <[email protected]>
Date: 2014-09-09T21:52:37Z
make config names more consistent with each other
commit 2aa03468db3971a11d07048628908074f2a0d3b3
Author: Rick Kilgore <[email protected]>
Date: 2014-09-09T23:03:34Z
do the time calc only once
do the due time calculation for a MessageRetryRecord when one is
created, not every time isReadyForRetry() is calculated
commit f4fa76c63efefa987ba124dd7dc8f895f9fe6dcc
Author: Rick Kilgore <[email protected]>
Date: 2014-09-09T23:06:56Z
silly fix
use member var, don't pass it
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
