[ https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726614#comment-16726614 ]
Stig Rohde Døssing commented on STORM-2359: ------------------------------------------- {quote}Acker will be totally clueless if all acks from downstream bolts are also lost for the same tuple tree.{quote} Yes. This is why the sync tuple is needed. If the spout executor doesn't time out the tree on its own, then we need a mechanism to deal with trees where the acker isn't aware of the tree due to message loss of the init and ack/fail messages. If we don't do a sync, the spout will never know that the acker isn't aware of the tree, and the tree will never fail. {quote} 1) Handle timeouts in SpoutExecutor. It sends a timeout msg to ACKer. *+Benefit+*: May not be any better than doing in ACKer. Do you have any thoughts about this ?{quote} I don't think there's any benefit to doing this over what we're doing now. Currently the spout executor times out the tuple, and the acker doesn't try to fail tuples on timeout. Instead it just quietly discards whatever information it has about a tree once the pending map rotates a few times. I'm not sure what we'd gain from the acker not rotating pending and relying on a timeout tuple instead. The benefit as I see it of moving the timeouts to the acker would be the ability to reset timeouts more frequently (e.g. on every ack) without increasing load on the spout executor, which we can't do if the spout is still handling the timeout. {quote} 2) Eliminate timeout communication between Spout & ACK. Let each does its own timeout.{quote} This is how it works now. As you note, there are some benefits to doing it this way. An additional benefit is that we can easily reason about the max time to fail a tuple on timeout, since the tuple will fail as soon as the spout rotates it out of pending. The drawbacks are: * Any time the timeout needs to be reset, a message needs to go to both the acker and the spout (current path is bolt -> acker -> spout) * Since resetting timeouts is reasonably expensive, we don't do it as part of regular acks, the user has to manually call collector.resetTimeout(). The benefit of moving timeouts entirely to the acker is that we can reset timeouts automatically on ack. This means that the tuple timeout becomes somewhat easier to work with, since tuples won't time out as long as an edge in the tree is getting acked occasionally. This is currently more of a loose idea, but in the long run I'd like to try adding a feature toggle for more aggressive automatic timeout resets. I'm not sure what the performance penalty would be, but it would be nice if the bolts could periodically reset the timeout for all queued/in progress tuples (tuples received by the worker, but not yet acked/failed by a collector). Doing this would eliminate the degenerate case you mention in the design doc, where some bolt is taking slightly too long to process a tuple, and the queued tuples time out, which causes the spout to reemit, which puts the reemits in queue behind the already timed out tuples. In some cases this can cause the topology to spend all its time processing timed out tuples, preventing it from making progress, even though each individual tuple could have been processed within the timeout. If this idea turns out to be possible to implement without adding a large overhead, it would add an extra drawback to timeouts in the spout: * We can't do automatic timeout resetting, since flooding the spout with reset messages is a bad idea. In particular we can't reset timeouts for messages that are sitting in bolt queues. The user can't reset these timeouts manually either. > Revising Message Timeouts > ------------------------- > > Key: STORM-2359 > URL: https://issues.apache.org/jira/browse/STORM-2359 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core > Affects Versions: 2.0.0 > Reporter: Roshan Naik > Assignee: Stig Rohde Døssing > Priority: Major > Attachments: STORM-2359.ods > > > A revised strategy for message timeouts is proposed here. > Design Doc: > > https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)