[
https://issues.apache.org/jira/browse/STORM-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021024#comment-15021024
]
ASF GitHub Bot commented on STORM-756:
--------------------------------------
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/897
STORM-756 Handle taskids response as soon as possible
This is the implementation of
https://github.com/apache/storm/pull/532#issuecomment-158578419
Please refer comment to see purpose of this PR.
Message's priority was heartbeat > taskids = BoltMsg, but in this PR, I
changed priority to heartbeat > taskids > BoltMsg.
In order to accomplish this, taskids and BoltMsgs use their own queue.
Queue for taskids is unbounded but queue for BoltMsgs is bounded (fixed size)
so that executor thread can be blocked and ABP comes into play.
Please note that BoltWriterRunnable polls message from both queues when it
is only available. It's for minimizing wait for taskids, but it can spin CPU so
CPU usage will be a bit higher than current.
I'd like to discuss about better / safer approach, for example, adding
potential lag via poll with timeout.
@revans2 It's alternative approach of #532. Please review and comment.
Thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-756-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/897.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #897
----
commit a16eca7e0b35c35be064a742f834add1e4f20879
Author: Jungtaek Lim <[email protected]>
Date: 2015-11-22T09:25:51Z
STORM-756 Handle taskids response ASAP
* create new queue which stores only taskids responses
** BoltReaderRunnable thread is no longer blocked by _pendingWrites.put()
* let BoltWriterRunnable sends messages with respecting priorities
** heartbeat > taskids > tuple
* set sleep time from multilang_test long enough
** that topology is activate and it processes some tuples for a while
commit d91c393f092ad2e050f14bc3c6ab9e9658545c26
Author: Jungtaek Lim <[email protected]>
Date: 2015-11-22T09:56:50Z
STORM-756 set default value of topology.shellbolt.max.pending
* Bounded _pendingWrites makes ShellBolt play well with ABP
commit 124f664af6d8cdf995ed4061b1f1a90c84e7c18e
Author: Jungtaek Lim <[email protected]>
Date: 2015-11-22T14:15:19Z
Remove unneeded comment
----
> [multilang] Introduce overflow control mechanism
> ------------------------------------------------
>
> Key: STORM-756
> URL: https://issues.apache.org/jira/browse/STORM-756
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-multilang
> Affects Versions: 0.10.0, 0.9.4, 0.11.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
>
> It's from STORM-738,
> https://issues.apache.org/jira/browse/STORM-738?focusedCommentId=14394106&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14394106
> A. ShellBolt side control
> We can modify ShellBolt to have sent tuple ids list, and stop sending tuples
> when list exceeds configured max value. In order to achieve this, subprocess
> should notify "tuple id is complete" to ShellBolt.
> * It introduces new commands for multi-lang, "proceed" (or better name)
> * ShellBolt stores in-progress-of-processing tuples list.
> * Its overhead could be big, subprocess should always notify to ShellBolt
> when any tuples are processed.
> B. subprocess side control
> We can modify subprocess to check pending queue after reading tuple.
> If it exceeds configured max value, subprocess can request "delay" to
> ShellBolt for slowing down.
> When ShellBolt receives "delay", BoltWriterRunnable should stop polling
> pending queue and continue polling later.
> How long ShellBolt wait for resending? Its unit would be "delay time" or
> "tuple count". I don't know which is better yet.
> * It introduces new commands for multi-lang, "delay" (or better name)
> * I don't think it would be introduced soon, but subprocess can request delay
> based on own statistics. (ex. pending tuple count * average tuple processed
> time for time unit, average pending tuple count for count unit)
> ** We can leave when and how much to request "delay" to user. User can make
> his/her own algorithm to control flooding.
> In my opinion B seems to more natural cause current issue is by subprocess
> side so it would be better to let subprocess overcome it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
