[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369774#comment-15369774 ] Hudson commented on YARN-3041: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-3041. Added the overall data model of timeline service next gen. (sjlee: rev c77f86bf637afd528c8e4604521eac91467a492b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/timelineservice/TestTimelineServiceRecords.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/ContainerEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/HierarchicalTimelineEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/FlowEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/ClusterEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineEntityType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineUser.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/ApplicationEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineMetric.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/package-info.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/ApplicationAttemptEntity.java > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Fix For: YARN-2928 > > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, > YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327030#comment-14327030
]
Hadoop QA commented on YARN-3041:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699612/YARN-3041.5.patch
against trunk revision 946456c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build///testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build///console
This message is automatically generated.
> [Data Model] create overall data objects of TS next gen
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
> YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch,
> YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327028#comment-14327028 ] Zhijie Shen commented on YARN-3041: --- Cool! Thanks for your review, Sangjin! I'll go ahead to commit the patch. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, > YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327019#comment-14327019 ] Sangjin Lee commented on YARN-3041: --- LGTM. Thanks for reflecting the latest feedback! I agree with your points for the most part. The update of the design doc is long overdue. I'll try to update the document to reflect all the changes that have taken place so far. We'll file more JIRAs if we need to adjust/update the data model as the work progresses. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, > YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326185#comment-14326185 ] Joep Rottinghuis commented on YARN-3041: I think version may have to be something more than a property on a flow. We need to be able to query by versions. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326183#comment-14326183 ] Joep Rottinghuis commented on YARN-3041: Some additional throughs: If we have the types strongly typed, do we need to call containers "YARN_CONTAINER" and "YARN_FLOW", or would we be able to capture more generic flows and containers with this as well ? Perhaps the framework used to run could be a property for the generic entity. I don't see what the advantage is to have the user set up the proper relationship. Why not make that part of the constructors and have protected methods to set up the hierarchy correctly ? Why introduce a chance to have this all set up strange ? I think the acceptable entity types for parent-child relationships can be setup in the enum itself. The enums would simply have methods on them and can take constructors. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326178#comment-14326178 ] Joep Rottinghuis commented on YARN-3041: Agreed with [~sjlee0] that we should use an enum to enumerate the timeline entity types. Not sure if we should directly use enums, or have TimelineEntity.type be interface TimelineEntityType and have an enum that implements that interface. The latter is more extensible later on (there could be other enums implementing the interface). On the other hand that makes things a bit harder to enumerate over, so perhaps that is overkill. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325520#comment-14325520 ] Vrushali C commented on YARN-3041: -- So, for my clarification (perhaps I may be missing this), where would a Flow object be used versus FlowRun? Put another way, which functionality would return (or require) a Flow object and which functionality would return (or require) a FlowRun? > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325492#comment-14325492 ] Vrushali C commented on YARN-3041: -- To add to my previous comment, this is the way I see it: A flow is uniquely identified by cluster, user, queue, flow name and run id. So these are metadata/attributes/class members of the flow class. FlowRun is not a class, run id is an attribute/member of a Flow class. An Application is a child of a Flow. There would also be an AggregatedFlow class which has members like startTime and endTime of aggregation etc. Similarly, user and queue are attributes of the Flow class. But AggregatedUser and AggregatedQueue are classes, which have aggregated information for that user (or queue) over a time range. Maybe I can give some examples of queries. For Flow: Example 1 : we query for “Give me all the runs of this flow that happened yesterday”, Say the flow ran 10 times yesterday. This should return a list of 10 flows, one flow object for each run. Each flow object in turn has a list of Applications. Example 2 : we query for “ How much did this flow take up on the cluster yesterday? “ Say the flow ran 10 times yesterday. This query should return an aggregated flow object which has the summation of all metrics from all the run of the flow yesterday. This aggregatedFlow now also has the startTime and endTime of aggregation as it’s members. (While we would allow for custom time ranges, for efficiency we would want to aggregate daily, weekly etc.) For User Example 1: Query: give me all flows that this user ran over this time range. Returns a list of such flows, one flow object for each individual run. Example 2: Query: give me how much this user consumed on the cluster during this time range. Would return an AggregatedUser object which has startTime and endTime of this aggregation and summations of metrics over that time range. Again, for aggregations, we would probably want to aggregate daily, weekly etc while allowing for custom ranges. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325477#comment-14325477 ] Zhijie Shen commented on YARN-3041: --- Another issue about having run id as the attribute of fll=ow: For example, there's a flow alice@pig_analyze_browsers, and two flow run 001 and 002. Only alice@pig_analyze_browsers will be used as the identifier (type = YARN_FLOW, id = alice@pig_analyze_browsers). We're not able to create two flow entities \[alice@pig_analyze_browsers, 001\] and \[alice@pig_analyze_browsers, 002\]. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325468#comment-14325468
]
Hadoop QA commented on YARN-3041:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699422/YARN-3041.4.patch
against trunk revision 57db50c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6656//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6656//console
This message is automatically generated.
> [Data Model] create overall data objects of TS next gen
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
> YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325467#comment-14325467 ] Zhijie Shen commented on YARN-3041: --- bq. I think the run id of a flow is an attribute of a flow, it need not be a separate class. So what's the difference between flow and flow run? On page.5 of the design doc v2, it seems that flow run is the child of flow. [~sjlee0], would you please comment? bq. Also, queue and user are timeline entities, I think if they may not require a class by themselves. Do you mean they need to extend timeline entity? I think in the new patch, they do it in this way. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325432#comment-14325432 ] Vrushali C commented on YARN-3041: -- I think the run id of a flow is an attribute of a flow, it need not be a separate class. Also, queue and user are timeline entities, I think if they may not require a class by themselves. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325431#comment-14325431
]
Zhijie Shen commented on YARN-3041:
---
Thanks for the feedback, Sangjin! See my response below. I've made a new patch
accordingly.
bq. In defining types, how about using enums (i.e. stronger types) as opposed
to strings? Have you considered enums?
Because entity#type is String. I thought of using enum, but we still need to
call toString() to compare with other types.
bq. In HierarchicalTimelineEntity.addChildren(), I think there is a bug.
Good catch. I forgot {{this.}}. It suppose to modify the member variable, not
param. Fix the same problem in TimelineEntity.
bq. Shouldn't TimelineUser and TimelineQueue extend TimelineEntity?
I used to think we will do aggregation based on user and queue basis, but am
not sure if we will really record the detailed information about user and
queue. Anyway, I made them extend entity to open the option in case we will do
it later. Then, accordingly, I need to change entity to contain user/queue id
instead of object, preventing nested marshal/unmarshal.
bq. Are we talking about the “single application” situation outside the context
of a flow?
Yeah, it's the single application case, but not just limit to it. For example,
long running services can run as an application on YARN. We need capture the
non-flow based data model too.
bq. So I don't think a flow can be a direct parent of an application. Only a
flow run can be one.
Sounds reasonable. I changed it accordingly.
> [Data Model] create overall data objects of TS next gen
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
> YARN-3041.3.patch, YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325390#comment-14325390 ] Sangjin Lee commented on YARN-3041: --- Apologies for not chiming in sooner. I have been held up by a number of things that are going on. The patch looks mostly good. Thanks [~zjshen] for putting up the patch and the proposal. I also agree that we should get the patch committed soon and adjust later if necessary. Having said that, I do have some quick high level comments, and hopefully they are mostly straightforward. - In defining types, how about using enums (i.e. stronger types) as opposed to strings? Have you considered enums? - In HierarchicalTimelineEntity.addChildren(), I think there is a bug. The *argument* "children" is being iterated upon, but within the for loop, it is accessed and mutated again. I'm pretty sure this will throw a ConcurrentModificationException. I think the intent is to interact with the *member variable* "children" inside the loop, right? - Shouldn't TimelineUser and TimelineQueue extend TimelineEntity? They can (and will) have things like metrics associated with them. - In what sense could the flow and the cluster be parents of the application entity? Are we talking about the “single application” situation outside the context of a flow? Especially, I’m not sure about the flow being a parent of an application. The flow is a *template*, and only a specific realization of a flow (i.e. flow run) can contain real applications. So I don't think a flow can be a direct parent of an application. Only a flow run can be one. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325347#comment-14325347
]
Robert Kanter commented on YARN-3041:
-
{quote}the relates/is related entities and parent/children entities become IDs,
we can not simply do the check. Caller needs to make sure the relationship be
properly set. At the server side, we need to do the check too. Thoughts?{quote}
That's true. Otherwise, we'd have to go look them up somehow. I think it's
fine to have the caller/server ensure this is correct. We can always revisit
this later as well if it becomes a problem.
+1
> [Data Model] create overall data objects of TS next gen
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
> YARN-3041.3.patch, YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325301#comment-14325301
]
Hadoop QA commented on YARN-3041:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699352/YARN-3041.3.patch
against trunk revision 685af8a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6655//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6655//console
This message is automatically generated.
> [Data Model] create overall data objects of TS next gen
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Zhijie Shen
> Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
> YARN-3041.3.patch, YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325288#comment-14325288 ] Li Lu commented on YARN-3041: - Looked at the patch briefly and in general it LGTM. I would agree that we should let something in to unblock all related issues. We can always trace further improvements in separate JIRAs and then relate them to here. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325190#comment-14325190 ] Vinod Kumar Vavilapalli commented on YARN-3041: --- bq. I'd like to propose the scope of this Jira: let's try to make the blueprint of the data model ready here, and treat it as a working/unstable version. During our development we can revisit the data model and fix the problems on demand. Makes sense. +1 for this, so as to unblock other dependent patches. Cursorily looked through the patch, it's good. I'll let Robert do the honors given he's been spending time reviewing this. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325178#comment-14325178 ] Zhijie Shen commented on YARN-3041: --- bq. Is it up to the calling code to make sure that if it sets entity A as the parent of entity B, that it also sets entity B as the child of entity A and vice versa? In the pervious patch. If we correlate two entities, it's easy to implement the logic in the data object itself to enforce the relation integrity. However, for the aforementioned reason, the relates/is related entities and parent/children entities become IDs, we can not simply do the check. Caller needs to make sure the relationship be properly set. At the server side, we need to do the check too. Thoughts? > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325161#comment-14325161 ] Robert Kanter commented on YARN-3041: - The new patch looks good. I agree that we should treat this as working/unstable; I think it's better to get this in sooner so that other JIRAs can be unblocked. So I'm fine if we leave some things a bit rough for now and refine them later. One last thing from me: - Is it up to the calling code to make sure that if it sets entity A as the parent of entity B, that it also sets entity B as the child of entity A and vice versa? Or should we take care of this internally? Same with isRelatedToEntities and relatesToEntities? > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325074#comment-14325074 ] Zhijie Shen commented on YARN-3041: --- I'd like to propose the scope of this Jira: let's try to make the blueprint of the data model ready here, and treat it as a working/unstable version. During our development we can revisit the data model and fix the problems on demand. > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
