[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174312#comment-16174312 ] Jongyoul Lee commented on YARN-6771: Sounds great!! Thanks!! > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee >Assignee: Jongyoul Lee > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771-6.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174300#comment-16174300 ] Hudson commented on YARN-6771: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12931 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12931/]) YARN-6771. Use classloader inside configuration class to make new (sjlee: rev 8b336632acad10e45d029596c5e3196e1857d891) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcClientFactoryPBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/factories/impl/pb/TestRpcServerFactoryPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/factories/impl/pb/TestRpcClientFactoryPBImpl.java > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771-6.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174281#comment-16174281 ] Sangjin Lee commented on YARN-6771: --- LGTM. Committing it shortly. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771-6.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170261#comment-16170261 ] Hadoop QA commented on YARN-6771: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 40s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 11s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-6771 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12887688/YARN-6771-6.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 914d4a3a2575 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a4f9c7c | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17498/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17498/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771-6.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170206#comment-16170206 ] Hadoop QA commented on YARN-6771: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} YARN-6771 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-6771 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12887686/YARN-6771-5.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17497/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771-4.patch, YARN-6771-5.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149681#comment-16149681 ] Junping Du commented on YARN-6771: -- Drop 2.8.2 in fix version. We should only use fix version when patch is committed. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111829#comment-16111829 ] Sangjin Lee commented on YARN-6771: --- Thanks for the update. Could you please look at the checkstyle violations? They are related to your changes. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110477#comment-16110477 ] Hadoop QA commented on YARN-6771: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 21s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 23 new + 21 unchanged - 0 fixed = 44 total (was 21) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6771 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879971/YARN-6771-3.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 785e4b26102f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f9139ac | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16664/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16664/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16664/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110436#comment-16110436 ] Jongyoul Lee commented on YARN-6771: I've updated new patch which include your comments. And I'll try to do wider test in my laptop. I'll let you know the result after finish it. Thanks for guiding me > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771-3.patch, > YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096454#comment-16096454 ] Sangjin Lee commented on YARN-6771: --- Thanks for linking the ticket. As for wider tests, I think it'd be great if you can run full YARN and mapreduce tests. If you have a working development environment, one could do {code} mvn clean install -DskipTests cd hadoop-yarn-project/hadoop-yarn mvn test -fn cd ../../hadoop-mapreduce-project mvn test -fn {code} Now, this is easier said than done. The reality is that this will take quite a while to run (around a couple of hours). And unfortunately I think there are some tests that are flaky or may fail depending on the environment. So this is not going to be exact science. The best effort is to run this on the current trunk, run it again with your patch, and see if there are "obvious" new failures that may be related to your patch. Let me know if that is something you can try. For setting up the dev environment, you can look at https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093124#comment-16093124 ] Jongyoul Lee commented on YARN-6771: I've linked ZEPPELIN-2040 as a related issue to it. And I'll fix {{RpcServerFactoryRBImpl}} too. Concerning test, I'm not sure how I can test it. Can you guide me how to test or write a integration test for it? This is my first PR for Hadoop side, so I don't know the rule yet. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093116#comment-16093116 ] Sangjin Lee commented on YARN-6771: --- Do you have a JIRA on the Zeppelin side that describes the actual issue? I suppose this is an issue when you need to load a class (that is only on your classpath) dynamically via {{Configuration}}? Could you please update your patch to include {{RpcServerFactoryPBImpl}} too? Although the client alone would address your issue, from the YARN perspective, it would not be ideal to have an asymmetric behavior. I think we also need to run a wider test suite than what jenkins runs out of the change analysis to ensure nothing changes as a result... > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092568#comment-16092568 ] Jongyoul Lee commented on YARN-6771: Thanks for the reply. I'm developing cluster manager feature in Apache Zeppelin. I've tried to use Yarn Client to communicate with Yarn cluster. Avoiding conflicts of dependencies, I wanted to use custom classloader for initializing yarn client in the Zeppelin's server process, but it failed because this implementation uses system classloader even I set the custom classloader in the Configuration. I changed my implementation to include hadoop libs into process directly but I think we need to change it to work correctly. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087770#comment-16087770 ] Sangjin Lee commented on YARN-6771: --- Thanks for the contribution [~jongyoul]. It appears that the same pattern exists in {{RpcServerFactoryPBImpl}} too. To be consistent, we should change both or neither. That said, these are pretty basic classes that pin all YARN RPC communications. I don't see the full history, but it appears that using a clean configuration to load these classes was intentional. Could you please explain in bit more detail why you'd need to use provide a specific {{Configuration}} instance to create RPC clients/servers? Is there a JIRA or issue that describes an issue on your end? > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771-2.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079657#comment-16079657 ] Hadoop QA commented on YARN-6771: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 17s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 11 new + 8 unchanged - 0 fixed = 19 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 49s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6771 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12876324/YARN-6771-2.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux b8bc1f652310 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 626d730 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16343/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16343/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16343/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Jongyoul Lee >
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079650#comment-16079650 ] Hadoop QA commented on YARN-6771: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color} | {color:red} YARN-6771 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-6771 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12876323/YARN-6771-1.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16342/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079646#comment-16079646 ] Jongyoul Lee commented on YARN-6771: I attached new patch with test case > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771-1.patch, YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079540#comment-16079540 ] Jongyoul Lee commented on YARN-6771: It's related to classloader and its usage. To make a test case, we need class that is not included in default class loader it means we need to make a class in order not to be included when running test. I'm not sure what the best way is, but one possible way is to make a class dynamically when starting test. May I use that way to test it? > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077821#comment-16077821 ] Hadoop QA commented on YARN-6771: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6771 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12876045/YARN-6771.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ef17a34be61c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 82cb2a6 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16324/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16324/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > Attachments: YARN-6771.patch > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses
[jira] [Commented] (YARN-6771) Use classloader inside configuration class to make new classes
[ https://issues.apache.org/jira/browse/YARN-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1601#comment-1601 ] Jongyoul Lee commented on YARN-6771: Could someone please help to review it? > Use classloader inside configuration class to make new classes > --- > > Key: YARN-6771 > URL: https://issues.apache.org/jira/browse/YARN-6771 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Jongyoul Lee > Fix For: 2.8.2 > > > While running {{RpcClientFactoryPBImpl.getClient}}, > {{RpcClientFactoryPBImpl}} uses {{localConf.getClassByName}}. But in case of > using custom classloader, we have to use {{conf.getClassByName}} because > custom classloader is already stored in {{Configuration}} class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org