[
https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043409#comment-16043409
]
Stig Rohde Døssing commented on STORM-2359:
-------------------------------------------
{quote}
The resets are managed internally by the ACKer bolt. The spout only gets
notified if the timeout expires or if tuple-tree is fully processed.
{quote}
How will this work? The current implementation has a pending map in both the
acker and the spout, which rotate every topology.message.timeout.secs. If the
acker doesn't forward reset requests to the spout, the spout will just expire
the tuple tree on its own when the message timeout has passed.
{quote}
That case would be more accurately classified as "progress is being made" ...
but slower than expected.
The case of 'progress is not being made' is when a worker that is processing
part of the tuple tree dies.
{quote}
Yes, you are right. But it is currently possible that the topology may degrade
to no progress being made even if each individual tuple could be processed
under the message timeout, because tuples can expire while queued and get
reemitted, where they can then be delayed by their own duplicates which are
ahead in the queue. For IRichBolts, this can be mitigated by the bolt being
written to accept and queue tuples internally, where the bolt can then reset
their timeouts manually if necessary, but for IBasicBolt this is not possible.
Just to give a concrete example, we had an IBasicBolt enrich tuples with some
database data. Most tuples were processed very quickly, but a few were slow.
Even the slow tuples never took longer than our message timeout individually.
We then had an instance where a bunch of slow tuples happened to come in on the
stream close to each other. The first few were processed before they expired,
but the rest expired while queued. The spout then reemitted the expired tuples,
and they got into the queue behind their own expired instances. Since the bolt
won't skip expired tuples, the freshly emitted tuples also expired, which
caused another reemit. This repeated until the topology was restarted so the
queues could be cleared.
> Revising Message Timeouts
> -------------------------
>
> Key: STORM-2359
> URL: https://issues.apache.org/jira/browse/STORM-2359
> Project: Apache Storm
> Issue Type: Sub-task
> Components: storm-core
> Affects Versions: 2.0.0
> Reporter: Roshan Naik
>
> A revised strategy for message timeouts is proposed here.
> Design Doc:
>
> https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
