[
https://issues.apache.org/jira/browse/STORM-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393496#comment-15393496
]
ASF GitHub Bot commented on STORM-2006:
---------------------------------------
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1594
STORM-2006 Storm metrics feature improvement: support per-worker level
metrics aggregation (1.x)
* SystemBolt handles task level metrics via two mode
** non-aggregate: same to previous, just applying expansion and pass to
MetricConsumerBolts
** aggregate: apply expansion, do aggregation, pass aggregated metrics to
MetricConsumerBolts
* all task level metrics should pass by SystemBolt within its worker
** it drops all connections between tasks and MetricsConsumerBolts
** only SystemBolts and MetricConsumerBolts will be connected
* move configurations: expandMapType and metricNameSeparator to global
** since SystemBolt needs to handle expansion when both worker level
aggregation and expandMapType are turned on
Manually tested via matrix of options : aggregate x expand
I'm not sure how to create unit tests playing with SystemBolt, but will try
to come up unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-2006-1.x
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1594.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1594
----
commit b2acd027768c30b71357a55a8eccd13a485089f3
Author: Jungtaek Lim <[email protected]>
Date: 2016-07-25T13:43:02Z
STORM-2006 Storm metrics feature improvement: support per-worker level
metrics aggregation
* SystemBolt handles task level metrics via two mode
** non-aggregate: same to previous, just applying expansion and pass to
MetricConsumerBolts
** aggregate: apply expansion, do aggregation, pass aggregated metrics to
MetricConsumerBolts
* all task level metrics should pass by SystemBolt within its worker
** it drops all connections between tasks and MetricsConsumerBolts
** only SystemBolts and MetricConsumerBolts will be connected
* move configurations: expandMapType and metricNameSeparator to global
** since SystemBolt needs to handle expansion when both worker level
aggregation and expandMapType are turned on
----
> Storm metrics feature improvement: support per-worker level metrics
> aggregation
> -------------------------------------------------------------------------------
>
> Key: STORM-2006
> URL: https://issues.apache.org/jira/browse/STORM-2006
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Affects Versions: 1.1.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
>
> Storm provides per-task level metrics which could be huge when topology has a
> number of tasks.
> Task level metric is useful for determining load balance between tasks, but
> it doesn't need to be time-series fashion.
> Before introducing topology level component like TopologyMaster for JStorm,
> we can utilize SystemBolt to aggregate task level metrics to per-worker level
> metrics.
> We should provide options and this feature should be turned off by default to
> keep backward compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
