[
https://issues.apache.org/jira/browse/STORM-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263672#comment-15263672
]
ASF GitHub Bot commented on STORM-1742:
---------------------------------------
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1379
[WIP] STORM-1742 More accurate 'complete latency'
**NOTE: need to do some performance tests cause Acker needs to work more,
and `time delta` should be delivered between Acker and Spout.**
* Acker computes 'complete latency' and sends back to Spout
* start time: Acker receiving ACK_INIT from Spout
* end time: Acker receiving ACK message which make the tuple tree
completed
* When Acker provides complete latency, Spout just uses this value to
record that latency
* exception case: tuple timed out - Spout doesn't receive failed
information from Acker
Btw, in fact Spout measures latency only acked tuple, not failed tuple. If
performance matters, we can get rid of using time delta with fail / reset
timeout cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-1742-1.x
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1379.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1379
----
commit 881b43c573cfb365d084da56b6d56bf768a93971
Author: Jungtaek Lim <[email protected]>
Date: 2016-04-28T02:25:51Z
STORM-1742 More accurate 'complete latency'
* Acker computes 'complete latency' and sends back to Spout
* start time: Acker receiving ACK_INIT from Spout
* end time: Acker receiving ACK message which make the tuple tree
completed
* When Acker provides complete latency, Spout just uses this value to
record that latency
* exception case: tuple timed out - Spout doesn't receive failed
information from Acker
----
> More accurate 'complete latency'
> --------------------------------
>
> Key: STORM-1742
> URL: https://issues.apache.org/jira/browse/STORM-1742
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
>
> I already initiated talking thread on dev@ list. Below is copy of the content
> in my mail.
> http://mail-archives.apache.org/mod_mbox/storm-dev/201604.mbox/%3CCAF5108gn=rskundfs7-sgy_pd-_prgj2hf2t5e5zppp-knd...@mail.gmail.com%3E
> While thinking about metrics improvements, I doubt how many users know that
> what 'exactly' is complete latency. In fact, it's somewhat complicated
> because additional waiting time could be added to complete latency because
> of single-thread model event loop of spout.
> Long running nextTuple() / ack() / fail() can affect complete latency but
> it's behind the scene. No latency information provided, and someone even
> didn't know about this characteristic. Moreover, calling nextTuple() could
> be skipped due to max spout waiting, which will make us harder to guess
> when avg. latency of nextTuple() will be provided.
> I think separation of threads (tuple handler to separate thread, as JStorm
> provides) would resolve the gap, but it requires our spout logic to be
> thread-safe, so I'd like to find workaround first.
> My sketched idea is let Acker decides end time for root tuple.
> There're two subsequent ways to decide start time for root tuple,
> 1. when Spout about to emit ACK_INIT to Acker (in other words, keep it as
> it is)
> - Acker sends ack / fail message to Spout with timestamp, and Spout
> calculates time delta
> - pros. : It's most accurate way since it respects the definition of
> 'complete latency'.
> - cons. : The sync of machine time between machines are very important.
> Milliseconds of precision would be required.
> 2. when Acker receives ACK_INIT from Spout
> - Acker calculates time delta itself, and sends ack / fail message to
> Spout with time delta
> - pros. : No requirement to sync the time between servers so strictly.
> - cons. : It doesn't contain the latency to send / receive ACK_INIT
> between Spout and Acker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
