[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369803#comment-15369803 ] Hudson commented on YARN-3125: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-3125. Made the distributed shell use timeline service next gen and (sjlee: rev d45ff878c4cb8b359abb17ecf09d24b6f862874c) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/aggregator/BaseAggregatorService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/aggregator/PerNodeAggregatorServer.java > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Fix For: YARN-2928 > > Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch, > YARN-3125_UT-022715.patch, YARN-3125v2.patch, YARN-3125v3.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340408#comment-14340408 ] Junping Du commented on YARN-3125: -- Thanks [~gtCarrera9] for adding a UT and [~zjshen] for review and commit it! > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Fix For: YARN-2928 > > Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch, > YARN-3125_UT-022715.patch, YARN-3125v2.patch, YARN-3125v3.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340354#comment-14340354 ] Zhijie Shen commented on YARN-3125: --- +1 for the new UT patch. Will commit the DS and the UT patch together > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch, > YARN-3125_UT-022715.patch, YARN-3125v2.patch, YARN-3125v3.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[
https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339715#comment-14339715
]
Zhijie Shen commented on YARN-3125:
---
Li, thanks for working on the patch. Looking at the test code of
TestDistributedShell. MiniYarnCluster will be started for each individual test
case. Therefore, we can potentially avoid conflict by configuration, and don't
need to hard code the service address. For those test cases about v1 timeline
service, you set enableAHS = true, while for the test cases about v2, you add
the aux service configuration. In this way, either v1 or v2 timeline service
will be set up, but not both. To distinguish the different test cases, you can
try the following thing:
{code}
@Rule public TestName name = new TestName();
{code}
Using the test name to switching the setup of MiniYarnCluster in setup().
Another minor issue. Instead of using
{code}
private static final String TIMELINE_AUX_SERVICE_CLASS =
"org.apache.hadoop.yarn.server.timelineservice.aggregator"
+ ".PerNodeAggregatorServer";
{code}
You can use {{PerNodeAggregatorServer.class.getName()}} directly.
> [Event producers] Change distributed shell to use new timeline service
> --
>
> Key: YARN-3125
> URL: https://issues.apache.org/jira/browse/YARN-3125
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Junping Du
> Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch,
> YARN-3125v2.patch, YARN-3125v3.patch
>
>
> We can start with changing distributed shell to use new timeline service once
> the framework is completed, in which way we can quickly verify the next gen
> is working fine end-to-end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339246#comment-14339246 ] Zhijie Shen commented on YARN-3125: --- Thanks for the patch, Junping! It looks good to me. Per offline discussion, we should add an integration test in TestDistributedShell. > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Attachments: YARN-3125.patch, YARN-3125v2.patch, YARN-3125v3.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[
https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337459#comment-14337459
]
Zhijie Shen commented on YARN-3125:
---
Hi Junping, thanks for updating the patch. Some more comments:
1. One side-by-side change is missed around the following code.
{code}
if(timelineClient != null) {
publishApplicationAttemptEvent(timelineClient, appAttemptID.toString(),
DSEvent.DS_APP_ATTEMPT_END, domainId, appSubmitterUgi);
}
{code}
2. Client needs to be changed too: taking CLI option and appending it as the
args of the command to run AM.
> [Event producers] Change distributed shell to use new timeline service
> --
>
> Key: YARN-3125
> URL: https://issues.apache.org/jira/browse/YARN-3125
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Junping Du
> Attachments: YARN-3125.patch, YARN-3125v2.patch
>
>
> We can start with changing distributed shell to use new timeline service once
> the framework is completed, in which way we can quickly verify the next gen
> is working fine end-to-end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336395#comment-14336395 ] Junping Du commented on YARN-3125: -- Thanks for comments, [~vinodkv] and [~zjshen]! Address both comments in v2 patch. > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Attachments: YARN-3125.patch, YARN-3125v2.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[
https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335798#comment-14335798
]
Zhijie Shen commented on YARN-3125:
---
My two cents: let's use {{createTimelineClient(ApplicationId appId)}} to
construct the client.
> [Event producers] Change distributed shell to use new timeline service
> --
>
> Key: YARN-3125
> URL: https://issues.apache.org/jira/browse/YARN-3125
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Junping Du
> Attachments: YARN-3125.patch
>
>
> We can start with changing distributed shell to use new timeline service once
> the framework is completed, in which way we can quickly verify the next gen
> is working fine end-to-end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335785#comment-14335785 ] Vinod Kumar Vavilapalli commented on YARN-3125: --- Quick comment: I think we should try not to disturb the old code much. Let's just add two separate independent code blocks without removing the old style event-push. > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Attachments: YARN-3125.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335400#comment-14335400 ] Junping Du commented on YARN-3125: -- Talked offline to Zhijie to take over this JIRA. > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
