[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351984#comment-16351984 ] ASF GitHub Bot commented on TWILL-254: -- Github user cbaenziger commented on the issue: https://github.com/apache/twill/pull/65 Thank you for the super fast code review @serranom! I think all comments have been addressed and the code should be complete now. Please let me know if you see anything else necessary. It is great working with Twill! Hopefully I can provide some other ideas as I work with it. > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twill issue #65: (TWILL-254) Update to use ContainerId.fromString
Github user cbaenziger commented on the issue: https://github.com/apache/twill/pull/65 Thank you for the super fast code review @serranom! I think all comments have been addressed and the code should be complete now. Please let me know if you see anything else necessary. It is great working with Twill! Hopefully I can provide some other ideas as I work with it. ---
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351925#comment-16351925 ] ASF GitHub Bot commented on TWILL-254: -- Github user cbaenziger commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165862926 --- Diff: twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java --- @@ -26,7 +26,7 @@ /** * Wrapper class for AMRMClient for Hadoop version 2.2 or greater. */ -public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { +public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { --- End diff -- If marked `final`, I can not extend it for [`Hadoop26YarnAMClient`](https://github.com/apache/twill/pull/65/files#diff-8c254de8ae00c6007495979dcb66a986R30)? > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user cbaenziger commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165862926 --- Diff: twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java --- @@ -26,7 +26,7 @@ /** * Wrapper class for AMRMClient for Hadoop version 2.2 or greater. */ -public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { +public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { --- End diff -- If marked `final`, I can not extend it for [`Hadoop26YarnAMClient`](https://github.com/apache/twill/pull/65/files#diff-8c254de8ae00c6007495979dcb66a986R30)? ---
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351924#comment-16351924 ] ASF GitHub Bot commented on TWILL-254: -- Github user cbaenziger commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165862857 --- Diff: twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java --- @@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) { this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf); } + @Override + private ContainerId containerIdLookup(String containerIdStr) { +return (ConverterUtils.toContainerId(containerIdStr)); --- End diff -- Thank you > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user cbaenziger commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165862857 --- Diff: twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java --- @@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) { this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf); } + @Override + private ContainerId containerIdLookup(String containerIdStr) { +return (ConverterUtils.toContainerId(containerIdStr)); --- End diff -- Thank you ---
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351923#comment-16351923 ] ASF GitHub Bot commented on TWILL-254: -- Github user cbaenziger commented on the issue: https://github.com/apache/twill/pull/65 I added a few omissions and corrected the parent `pom.xml` from my original submission. Now the new code is actually being used and I have updated the default target to Hadoop 2.6 and removed the generic `hadoop.version` as suggested. > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twill issue #65: (TWILL-254) Update to use ContainerId.fromString
Github user cbaenziger commented on the issue: https://github.com/apache/twill/pull/65 I added a few omissions and corrected the parent `pom.xml` from my original submission. Now the new code is actually being used and I have updated the default target to Hadoop 2.6 and removed the generic `hadoop.version` as suggested. ---
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351922#comment-16351922 ] ASF GitHub Bot commented on TWILL-254: -- Github user cbaenziger commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165862673 --- Diff: pom.xml --- @@ -680,9 +680,9 @@ -hadoop-2.5 +hadoop-2.6 -2.5.1 +2.6.5 --- End diff -- I didn't see any code specific to 2.5 that 2.3 did not provide? > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user cbaenziger commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165862673 --- Diff: pom.xml --- @@ -680,9 +680,9 @@ -hadoop-2.5 +hadoop-2.6 -2.5.1 +2.6.5 --- End diff -- I didn't see any code specific to 2.5 that 2.3 did not provide? ---
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351800#comment-16351800 ] ASF GitHub Bot commented on TWILL-254: -- Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845936 --- Diff: twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java --- @@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) { this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf); } + @Override + private ContainerId containerIdLookup(String containerIdStr) { +return (ConverterUtils.toContainerId(containerIdStr)); --- End diff -- this should be protected not private > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351797#comment-16351797 ] ASF GitHub Bot commented on TWILL-254: -- Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845875 --- Diff: pom.xml --- @@ -680,9 +680,9 @@ -hadoop-2.5 +hadoop-2.6 -2.5.1 +2.6.5 --- End diff -- why are you eliminating the hadoop-2.5 profile? > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351798#comment-16351798 ] ASF GitHub Bot commented on TWILL-254: -- Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845855 --- Diff: pom.xml --- @@ -171,7 +171,7 @@ 4.11 3.2 1.5 -[2.0.2-alpha,2.3.0] +2.7.2 target/hadoop20-classes --- End diff -- is there any reason to keep this property at all? seems better to just have it defined in each profile as necessary. > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString
[ https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351799#comment-16351799 ] ASF GitHub Bot commented on TWILL-254: -- Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845929 --- Diff: twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java --- @@ -26,7 +26,7 @@ /** * Wrapper class for AMRMClient for Hadoop version 2.2 or greater. */ -public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { +public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { --- End diff -- why remove final? > Update to use ContainerId.fromString > > > Key: TWILL-254 > URL: https://issues.apache.org/jira/browse/TWILL-254 > Project: Apache Twill > Issue Type: Bug > Components: yarn >Affects Versions: 0.12.0 >Reporter: Clay B. >Priority: Trivial > > Hello, trying to use 0.12 or master, I get the following on a Hortonworks > 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by > specifically building for Hadoop 2.6+ via {{mvn package > -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) > I have a patch to move us away from the deprecated (see YARN-1942) call in > the stack trace anyhow and provide Hadoop 2.6 as a supported profile. > {code:java} > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73) > Caused by: java.lang.RuntimeException: > java.lang.reflect.InvocationTargetException > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85) > at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71) > ... 5 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > ... 7 more > Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > container_e164_1516738752188_0024_01_01 > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > at > org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83) > at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65) > at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34) > ... 12 more > Caused by: java.lang.NumberFormatException: For input string: "e164" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:589) > at java.lang.Long.parseLong(Long.java:631) > at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > ... 15 more > {code} > For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed > container ID formats and a new API call came out > [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)]. > Meanwhile, the container ID format change was not reflected, of course, in > older versions of Hadoop as clarified in YARN-3575. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845875 --- Diff: pom.xml --- @@ -680,9 +680,9 @@ -hadoop-2.5 +hadoop-2.6 -2.5.1 +2.6.5 --- End diff -- why are you eliminating the hadoop-2.5 profile? ---
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845855 --- Diff: pom.xml --- @@ -171,7 +171,7 @@ 4.11 3.2 1.5 -[2.0.2-alpha,2.3.0] +2.7.2 target/hadoop20-classes --- End diff -- is there any reason to keep this property at all? seems better to just have it defined in each profile as necessary. ---
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845936 --- Diff: twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java --- @@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) { this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf); } + @Override + private ContainerId containerIdLookup(String containerIdStr) { +return (ConverterUtils.toContainerId(containerIdStr)); --- End diff -- this should be protected not private ---
[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString
Github user serranom commented on a diff in the pull request: https://github.com/apache/twill/pull/65#discussion_r165845929 --- Diff: twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java --- @@ -26,7 +26,7 @@ /** * Wrapper class for AMRMClient for Hadoop version 2.2 or greater. */ -public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { +public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient { --- End diff -- why remove final? ---