[
https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237602#comment-15237602
]
Sangjin Lee commented on YARN-3816:
-----------------------------------
We discussed the cases where we may need to support adding more info for the
metrics on YARN-4053. Especially see [this
comment|https://issues.apache.org/jira/browse/YARN-4053?focusedCommentId=14994603&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14994603]
(although going over the full discussion is informative). The conclusion was
that it would be good not to store additional metadata as column pre- or
post-fixes due to the complications mentioned in YARN-4053. If we can find a
way to avoid that here, it would be ideal. If this is to support offline
aggregation, options like separate configuration were also discussed.
If we end up storing that metadata in HBase, one thing we should *definitely*
avoid is the need to read it back to do any writes. We're ruling out doing
read-then-write as a principle, otherwise it would open up a world of pain in
terms of performance as well as correctness.
> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> ----------------------------------------------------------------------------
>
> Key: YARN-3816
> URL: https://issues.apache.org/jira/browse/YARN-3816
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Junping Du
> Assignee: Li Lu
> Labels: yarn-2928-1st-milestone
> Attachments: Application Level Aggregation of Timeline Data.pdf,
> YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch,
> YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch,
> YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.1.patch,
> YARN-3816-YARN-2928-v3.patch, YARN-3816-YARN-2928-v4.patch,
> YARN-3816-YARN-2928-v5.patch, YARN-3816-feature-YARN-2928.v4.1.patch,
> YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch
>
>
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include:
> resource (CPU, Memory) consumption across all containers, number of
> containers launched/completed/failed, etc. We need this for apps while they
> are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be
> aggregated to show details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based
> on Application-level aggregations rather than raw entity-level data as much
> less raws need to scan (with filter out non-aggregated entities, like:
> events, configurations, etc.).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
