[
https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668610#comment-15668610
]
Sangjin Lee commented on YARN-5814:
-----------------------------------
Thanks [~BINGXUE QIU] for adding more details. It sounds like a good starting
point. A couple of points.
It would be great to see some more details on the reader side of things. In the
case of HBase/Phoenix, more sophisticated analytical queries are going to be
based on the Phoenix (SQL) schema. In case of Druid, it would be interesting to
see some examples being served right out of the same schema.
Also, with readers we do a fair amount of predicate (filter) pushdown. We have
the set of timeline service filters which then get translated to HBase filters
and are pushed down to the queries. Have you looked at what kind of predicate
pushdown would be feasible in case of Druid?
I'd also like to bring your attention to the entity id prefix we just
introduced (YARN-5715) to solve the problem of more natural sort/selection
order. The lexicographical sort order based on the entity id's is going to fall
short of the expectation, thus the reason for introducing the entity id prefix.
You might want to take a look at that, and see how that would translate to the
Druid schema.
I'll comment more if I have more things to think about.
> Add druid as storage backend in YARN Timeline Service
> ------------------------------------------------------
>
> Key: YARN-5814
> URL: https://issues.apache.org/jira/browse/YARN-5814
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
> Affects Versions: 3.0.0-alpha2
> Reporter: Bingxue Qiu
> Attachments: Add-Druid-in-YARN-Timeline-Service.pdf
>
>
> h3. Introduction
> I propose to add druid as storage backend in YARN Timeline Service.
> We run more than 6000 applications and generate 450 million metrics daily in
> Alibaba Clusters with thousands of nodes. We need to collect and store
> meta/events/metrics data, online analyze the utilization reports of various
> dimensions and display the trends of allocation/usage resources for cluster
> by joining and aggregating data. It helps us to manage and optimize the
> cluster by tracking resource utilization.
> To achieve our goal we have changed to use druid as the storage instead of
> HBase and have achieved sub-second OLAP performance in our production
> environment for few months.
> h3. Analysis
> Currently YARN Timeline Service only supports aggregating metrics at a) flow
> level by FlowRunCoprocessor and b) application level metrics aggregating by
> AppLevelTimelineCollector, offline (time-based periodic) aggregation for
> flows/users/queues for reporting and analysis is planned but not yet
> implemented. YARN Timeline Service chooses Apache HBase as the primary
> storage backend. As we all know that HBase doesn't fit for OLAP.
> For arbitrary exploration of data,such as online analyze the utilization
> reports of various dimensions(Queue,Flow,Users,Application,CPU,Memory) by
> joining and aggregating data, Druid's custom column format enables ad-hoc
> queries without pre-computation. The format also enables fast scans on
> columns, which is important for good aggregation performance.
> To achieve our goal that support to online analyze the utilization reports of
> various dimensions, display the variation trends of allocation/usage
> resources for cluster, and arbitrary exploration of data, we propose to add
> druid storage and implement DruidWriter /DruidReader in YARN Timeline Service.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
