[
https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888738#comment-13888738
]
Vinod Kumar Vavilapalli commented on YARN-1530:
-----------------------------------------------
Thanks for the comments everyone. Some responses follow:
Lohit
bq. Yes, proxy server inside library, but only in AM not containers.
Yes, that's a good idea and inline with what I might have hinted in the doc.
When containers also start writing out events, we need to have these
intermediate aggregators for scalability. I was thinking NodeManagers but your
idea of AMs is better. If we make everyone only depend on a library, the
underlying protocol can be either RPC or REST together with a RPC server or a
proxy like you mentioned. If REST is itself the first-class API, yes,
proxy-server is the way.
Bobby,
bq. I am also a bit nervous about using the history service for recovery or as
a backend for the current MR APIs if we have a pub/sub system as a link between
the applications and the history service.
Agreed. This is a problem that exists even today with MR apps in a way with
HDFS acting as a pubsub system. We've seen the corner cases you are hinting at
even with the HDFS based history files. I think we now understand enough about
these edge cases. I haven't yet made the jump to making these events be the
base for recovery, but points noted for when we wish to.
Chris,
bq. 1. Is the expectation that people will be able to use this for INFO, WARN,
ERROR type application logging?
Application logging is not supposed to be through this channel. This feature is
fundamentally for meta-event information - frankly, the kind we log in MR
JobHistory files. We already have the ability for containers to write logs to a
local log-directory and the corresponding features for aggregation, rendering.
This is a high throughput path that IMO cannot be sustained by the solution of
this JIRA. I'll make it clear in the doc.
bq. 2. Regarding app-specific UIs, is it going to be possible to embed
app-specific UIs with YARN's UI, instead of having to run an app-specific
web-ui? There is some mention of JS UIs, but it's a little unclear whether this
would be embedded in YARN, or served from "somewhere" (to quote the docs. If
it's served from the RM (or some other web-ui in YARN), will it be up to ops to
decide which libraries are embedded, or up to
I thought I wrote it clearly, will have to reread and edit if necessary. The
idea is to either let users host their UI's elsewhere or give an ability for
admins to install UIs (that they have vetted for stability/security etc) in the
HIstoryServer itself.
bq. 3. What are the planned "out of the box implementations" for both the
storage and transport layers? REST+LevelDB+HBase? Are Flume and Kafka
implementations expected to happen outside of the YARN project?
We are planning Library(wrapping REST)+LevelDB+LocalFS for smaller deployments
and out of the box. We will then move to working on Library(wrapping
REST)+LevelDB+HBase for larger deployments.
The discussion of Flume/Kafka is related to the event-aggregation which we
currently are doing via a simple REST put to a web-service. Whether/when we
move to those implementations will depend on the urgency with which we run into
issues mentioned in the doc with REST APIs. But it's open at this point of time.
> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>
> Key: YARN-1530
> URL: https://issues.apache.org/jira/browse/YARN-1530
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Vinod Kumar Vavilapalli
> Attachments: application timeline design-20140108.pdf, application
> timeline design-20140116.pdf, application timeline design-20140130.pdf
>
>
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework
> data all by itself as YARN doesn't have a common solution. This JIRA attempts
> to solve the storage, management and serving of per-framework data from
> various applications, both running and finished. The aim is to change YARN to
> collect and store data in a generic manner with plugin points for frameworks
> to do their own thing w.r.t interpretation and serving.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
