[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[ https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451040#comment-16451040 ] Hudson commented on YARN-7250: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) YARN-7250. Update Shared cache client api to use URLs. (xyao: rev 5f494fc3d9ecbedc8999aded3acb4b9aebc9c61c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestSharedCacheClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/SharedCacheClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/SharedCacheClient.java > Update Shared cache client api to use URLs > -- > > Key: YARN-7250 > URL: https://issues.apache.org/jira/browse/YARN-7250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7250-trunk-001.patch > > > We should make the SharedCacheClient use api more consistent with other YARN > api methods. We can do this by doing two things: > # Update the SharedCacheClient#use api so that it returns a URL instead of a > Path. Currently yarn developers have to convert the path to a URL when > creating a LocalResources. It would be much smoother if they could just use a > URL passed to them by the shared cache client. > # Remove the portion of the client that deals with fragments as this is not > consistent with the rest of YARN. This functionality is bleeding in from the > MapReduce layer, which uses fragments to keep track of destination file > names. YARN's api does not use fragments. Instead the ContainerLaunchContext > expects a Map localResources, where the strings are > the destination file names. We should let the YARN application handle > destination file names however it wants instead of pushing this into the > shared cache api. Additionally, fragments are a clunky way to handle this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[ https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185066#comment-16185066 ] Hudson commented on YARN-7250: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12993 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12993/]) YARN-7250. Update Shared cache client api to use URLs. (ctrezzo: rev c114da5e64d14b1d9e614081c4171ea0391cb1aa) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/SharedCacheClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/SharedCacheClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestSharedCacheClientImpl.java > Update Shared cache client api to use URLs > -- > > Key: YARN-7250 > URL: https://issues.apache.org/jira/browse/YARN-7250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7250-trunk-001.patch > > > We should make the SharedCacheClient use api more consistent with other YARN > api methods. We can do this by doing two things: > # Update the SharedCacheClient#use api so that it returns a URL instead of a > Path. Currently yarn developers have to convert the path to a URL when > creating a LocalResources. It would be much smoother if they could just use a > URL passed to them by the shared cache client. > # Remove the portion of the client that deals with fragments as this is not > consistent with the rest of YARN. This functionality is bleeding in from the > MapReduce layer, which uses fragments to keep track of destination file > names. YARN's api does not use fragments. Instead the ContainerLaunchContext > expects a Map localResources, where the strings are > the destination file names. We should let the YARN application handle > destination file names however it wants instead of pushing this into the > shared cache api. Additionally, fragments are a clunky way to handle this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[ https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183463#comment-16183463 ] Chris Trezzo commented on YARN-7250: Thank you [~vrushalic] for the review! I will wait until tomorrow to commit, just in case there are any other comments. Otherwise, I plan to commit to trunk, branch-3.0 and branch-2. > Update Shared cache client api to use URLs > -- > > Key: YARN-7250 > URL: https://issues.apache.org/jira/browse/YARN-7250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: YARN-7250-trunk-001.patch > > > We should make the SharedCacheClient use api more consistent with other YARN > api methods. We can do this by doing two things: > # Update the SharedCacheClient#use api so that it returns a URL instead of a > Path. Currently yarn developers have to convert the path to a URL when > creating a LocalResources. It would be much smoother if they could just use a > URL passed to them by the shared cache client. > # Remove the portion of the client that deals with fragments as this is not > consistent with the rest of YARN. This functionality is bleeding in from the > MapReduce layer, which uses fragments to keep track of destination file > names. YARN's api does not use fragments. Instead the ContainerLaunchContext > expects a Map localResources, where the strings are > the destination file names. We should let the YARN application handle > destination file names however it wants instead of pushing this into the > shared cache api. Additionally, fragments are a clunky way to handle this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[ https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183461#comment-16183461 ] Vrushali C commented on YARN-7250: -- +1 Patch LGTM with the following understanding: - The interface itself is marked Unstable so it should be okay to change it. - Any documentation updates to be handled in https://issues.apache.org/jira/browse/YARN-2960 - Purely a client side change, the protobuf for the rpc isn't changing - For MapReduce to use this, work is in progress at MAPREDUCE-5951 > Update Shared cache client api to use URLs > -- > > Key: YARN-7250 > URL: https://issues.apache.org/jira/browse/YARN-7250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: YARN-7250-trunk-001.patch > > > We should make the SharedCacheClient use api more consistent with other YARN > api methods. We can do this by doing two things: > # Update the SharedCacheClient#use api so that it returns a URL instead of a > Path. Currently yarn developers have to convert the path to a URL when > creating a LocalResources. It would be much smoother if they could just use a > URL passed to them by the shared cache client. > # Remove the portion of the client that deals with fragments as this is not > consistent with the rest of YARN. This functionality is bleeding in from the > MapReduce layer, which uses fragments to keep track of destination file > names. YARN's api does not use fragments. Instead the ContainerLaunchContext > expects a Map localResources, where the strings are > the destination file names. We should let the YARN application handle > destination file names however it wants instead of pushing this into the > shared cache api. Additionally, fragments are a clunky way to handle this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[ https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183443#comment-16183443 ] Chris Trezzo commented on YARN-7250: Also to clarify, this is a client side only change. The protobuf/rpc between the client and the SCM is staying the same. > Update Shared cache client api to use URLs > -- > > Key: YARN-7250 > URL: https://issues.apache.org/jira/browse/YARN-7250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: YARN-7250-trunk-001.patch > > > We should make the SharedCacheClient use api more consistent with other YARN > api methods. We can do this by doing two things: > # Update the SharedCacheClient#use api so that it returns a URL instead of a > Path. Currently yarn developers have to convert the path to a URL when > creating a LocalResources. It would be much smoother if they could just use a > URL passed to them by the shared cache client. > # Remove the portion of the client that deals with fragments as this is not > consistent with the rest of YARN. This functionality is bleeding in from the > MapReduce layer, which uses fragments to keep track of destination file > names. YARN's api does not use fragments. Instead the ContainerLaunchContext > expects a Map localResources, where the strings are > the destination file names. We should let the YARN application handle > destination file names however it wants instead of pushing this into the > shared cache api. Additionally, fragments are a clunky way to handle this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[ https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179819#comment-16179819 ] Chris Trezzo commented on YARN-7250: The TestNMClient failure is happening on trunk as well, and is unrelated to this patch. The same goes for the TestAMRMClient timeout. The patch should be good to go. The patch is technically an incompatible change, but this api is marked unstable and this feature is still in an alpha state, so there should be no issue. My intention is to, pending review, check this into trunk, branch-3.0 and branch-2. > Update Shared cache client api to use URLs > -- > > Key: YARN-7250 > URL: https://issues.apache.org/jira/browse/YARN-7250 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Minor > Attachments: YARN-7250-trunk-001.patch > > > We should make the SharedCacheClient use api more consistent with other YARN > api methods. We can do this by doing two things: > # Update the SharedCacheClient#use api so that it returns a URL instead of a > Path. Currently yarn developers have to convert the path to a URL when > creating a LocalResources. It would be much smoother if they could just use a > URL passed to them by the shared cache client. > # Remove the portion of the client that deals with fragments as this is not > consistent with the rest of YARN. This functionality is bleeding in from the > MapReduce layer, which uses fragments to keep track of destination file > names. YARN's api does not use fragments. Instead the ContainerLaunchContext > expects a Map localResources, where the strings are > the destination file names. We should let the YARN application handle > destination file names however it wants instead of pushing this into the > shared cache api. Additionally, fragments are a clunky way to handle this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7250) Update Shared cache client api to use URLs
[
https://issues.apache.org/jira/browse/YARN-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179656#comment-16179656
]
Hadoop QA commented on YARN-7250:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 20s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestNMClient |
| Timed out junit tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7250 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12888926/YARN-7250-trunk-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0fb59ac33e3c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e928ee5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17625/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17625/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17625/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Update Shared cache client api to use URLs
> --
>
> Key: YARN-7250
> URL: https://issues.apache.org/jira/browse/YARN-7250
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>
