[
https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15252716#comment-15252716
]
Li Lu commented on YARN-3816:
-----------------------------
Thanks [~sjlee0]. Some quick note on the design: the goal here is to let each
concrete types of timeline collectors to define their own skip types. The
challenge was the updateAggregateStatus method is static so that we can provide
the aggregateEntities static method for offline aggregations. This limits the
ability to customize the "skip set" for each TimelineCollector's subclasses (no
method override for static methods). One solution is to use the strategy
pattern to let each timeline collector decides its own set of skipped types,
but I do not want each instance of timeline collectors to hold one skip set
since they should be the same for the same class. Therefore, I'm making the
getEntityTypesSkipAggregation method to be an instance method, but for both
TimelineCollectors and AppLevelCollectors, they can simply return the class
level skip set. The two static sets (entityTypesSkipAggregation) just happen to
have the same name in the two classes, but they're not interfering with each
other.
Not sure if this is clear enough, but any suggestions would be helpful. Thanks!
> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> ----------------------------------------------------------------------------
>
> Key: YARN-3816
> URL: https://issues.apache.org/jira/browse/YARN-3816
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Junping Du
> Assignee: Li Lu
> Labels: yarn-2928-1st-milestone
> Attachments: Application Level Aggregation of Timeline Data.pdf,
> YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch,
> YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch,
> YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.1.patch,
> YARN-3816-YARN-2928-v3.patch, YARN-3816-YARN-2928-v4.patch,
> YARN-3816-YARN-2928-v5.patch, YARN-3816-YARN-2928-v6.patch,
> YARN-3816-YARN-2928-v7.patch, YARN-3816-YARN-2928-v8.patch,
> YARN-3816-YARN-2928-v9.patch, YARN-3816-feature-YARN-2928.v4.1.patch,
> YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch
>
>
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include:
> resource (CPU, Memory) consumption across all containers, number of
> containers launched/completed/failed, etc. We need this for apps while they
> are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be
> aggregated to show details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based
> on Application-level aggregations rather than raw entity-level data as much
> less raws need to scan (with filter out non-aggregated entities, like:
> events, configurations, etc.).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
