GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/897
STORM-756 Handle taskids response as soon as possible
This is the implementation of
https://github.com/apache/storm/pull/532#issuecomment-158578419
Please refer comment to see purpose of this PR.
Message's priority was heartbeat > taskids = BoltMsg, but in this PR, I
changed priority to heartbeat > taskids > BoltMsg.
In order to accomplish this, taskids and BoltMsgs use their own queue.
Queue for taskids is unbounded but queue for BoltMsgs is bounded (fixed size)
so that executor thread can be blocked and ABP comes into play.
Please note that BoltWriterRunnable polls message from both queues when it
is only available. It's for minimizing wait for taskids, but it can spin CPU so
CPU usage will be a bit higher than current.
I'd like to discuss about better / safer approach, for example, adding
potential lag via poll with timeout.
@revans2 It's alternative approach of #532. Please review and comment.
Thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-756-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/897.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #897
----
commit a16eca7e0b35c35be064a742f834add1e4f20879
Author: Jungtaek Lim <[email protected]>
Date: 2015-11-22T09:25:51Z
STORM-756 Handle taskids response ASAP
* create new queue which stores only taskids responses
** BoltReaderRunnable thread is no longer blocked by _pendingWrites.put()
* let BoltWriterRunnable sends messages with respecting priorities
** heartbeat > taskids > tuple
* set sleep time from multilang_test long enough
** that topology is activate and it processes some tuples for a while
commit d91c393f092ad2e050f14bc3c6ab9e9658545c26
Author: Jungtaek Lim <[email protected]>
Date: 2015-11-22T09:56:50Z
STORM-756 set default value of topology.shellbolt.max.pending
* Bounded _pendingWrites makes ShellBolt play well with ABP
commit 124f664af6d8cdf995ed4061b1f1a90c84e7c18e
Author: Jungtaek Lim <[email protected]>
Date: 2015-11-22T14:15:19Z
Remove unneeded comment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
