[
https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726614#comment-16726614
]
Stig Rohde Døssing commented on STORM-2359:
---
{quote}Acker will be totally clueless if all acks from downstream bolts are
also lost for the same tuple tree.{quote}
Yes. This is why the sync tuple is needed. If the spout executor doesn't time
out the tree on its own, then we need a mechanism to deal with trees where the
acker isn't aware of the tree due to message loss of the init and ack/fail
messages. If we don't do a sync, the spout will never know that the acker isn't
aware of the tree, and the tree will never fail.
{quote} 1) Handle timeouts in SpoutExecutor. It sends a timeout msg to ACKer.
*+Benefit+*: May not be any better than doing in ACKer. Do you have any
thoughts about this ?{quote}
I don't think there's any benefit to doing this over what we're doing now.
Currently the spout executor times out the tuple, and the acker doesn't try to
fail tuples on timeout. Instead it just quietly discards whatever information
it has about a tree once the pending map rotates a few times. I'm not sure what
we'd gain from the acker not rotating pending and relying on a timeout tuple
instead. The benefit as I see it of moving the timeouts to the acker would be
the ability to reset timeouts more frequently (e.g. on every ack) without
increasing load on the spout executor, which we can't do if the spout is still
handling the timeout.
{quote} 2) Eliminate timeout communication between Spout & ACK. Let each does
its own timeout.{quote}
This is how it works now. As you note, there are some benefits to doing it this
way. An additional benefit is that we can easily reason about the max time to
fail a tuple on timeout, since the tuple will fail as soon as the spout rotates
it out of pending. The drawbacks are:
* Any time the timeout needs to be reset, a message needs to go to both the
acker and the spout (current path is bolt -> acker -> spout)
* Since resetting timeouts is reasonably expensive, we don't do it as part of
regular acks, the user has to manually call collector.resetTimeout().
The benefit of moving timeouts entirely to the acker is that we can reset
timeouts automatically on ack. This means that the tuple timeout becomes
somewhat easier to work with, since tuples won't time out as long as an edge in
the tree is getting acked occasionally.
This is currently more of a loose idea, but in the long run I'd like to try
adding a feature toggle for more aggressive automatic timeout resets. I'm not
sure what the performance penalty would be, but it would be nice if the bolts
could periodically reset the timeout for all queued/in progress tuples (tuples
received by the worker, but not yet acked/failed by a collector). Doing this
would eliminate the degenerate case you mention in the design doc, where some
bolt is taking slightly too long to process a tuple, and the queued tuples time
out, which causes the spout to reemit, which puts the reemits in queue behind
the already timed out tuples. In some cases this can cause the topology to
spend all its time processing timed out tuples, preventing it from making
progress, even though each individual tuple could have been processed within
the timeout.
If this idea turns out to be possible to implement without adding a large
overhead, it would add an extra drawback to timeouts in the spout:
* We can't do automatic timeout resetting, since flooding the spout with reset
messages is a bad idea. In particular we can't reset timeouts for messages that
are sitting in bolt queues. The user can't reset these timeouts manually either.
> Revising Message Timeouts
> -
>
> Key: STORM-2359
> URL: https://issues.apache.org/jira/browse/STORM-2359
> Project: Apache Storm
> Issue Type: Sub-task
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Roshan Naik
>Assignee: Stig Rohde Døssing
>Priority: Major
> Attachments: STORM-2359.ods
>
>
> A revised strategy for message timeouts is proposed here.
> Design Doc:
>
> https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
