[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351984#comment-16351984
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
Thank you for the super fast code review @serranom! I think all comments 
have been addressed and the code should be complete now. Please let me know if 
you see anything else necessary. It is great working with Twill! Hopefully I 
can provide some other ideas as I work with it.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] twill issue #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread cbaenziger
Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
Thank you for the super fast code review @serranom! I think all comments 
have been addressed and the code should be complete now. Please let me know if 
you see anything else necessary. It is great working with Twill! Hopefully I 
can provide some other ideas as I work with it.


---


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351925#comment-16351925
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862926
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

If marked `final`, I can not extend it for 
[`Hadoop26YarnAMClient`](https://github.com/apache/twill/pull/65/files#diff-8c254de8ae00c6007495979dcb66a986R30)?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread cbaenziger
Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862926
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

If marked `final`, I can not extend it for 
[`Hadoop26YarnAMClient`](https://github.com/apache/twill/pull/65/files#diff-8c254de8ae00c6007495979dcb66a986R30)?


---


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351924#comment-16351924
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862857
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

Thank you


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread cbaenziger
Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862857
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

Thank you


---


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351923#comment-16351923
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
I added a few omissions and corrected the parent  `pom.xml` from my 
original submission. Now the new code is actually being used and I have updated 
the default target to Hadoop 2.6 and removed the generic `hadoop.version` as 
suggested.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] twill issue #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread cbaenziger
Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
I added a few omissions and corrected the parent  `pom.xml` from my 
original submission. Now the new code is actually being used and I have updated 
the default target to Hadoop 2.6 and removed the generic `hadoop.version` as 
suggested.


---


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351922#comment-16351922
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862673
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

I didn't see any code specific to 2.5 that 2.3 did not provide?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread cbaenziger
Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862673
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

I didn't see any code specific to 2.5 that 2.3 did not provide?


---


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351800#comment-16351800
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845936
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

this should be protected not private


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351797#comment-16351797
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845875
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

why are you eliminating the hadoop-2.5 profile?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351798#comment-16351798
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845855
  
--- Diff: pom.xml ---
@@ -171,7 +171,7 @@
 4.11
 3.2
 1.5
-[2.0.2-alpha,2.3.0]
+2.7.2
 target/hadoop20-classes
--- End diff --

is there any reason to keep this property at all?  seems better to just 
have it defined in each profile as necessary.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351799#comment-16351799
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845929
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

why remove final?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread serranom
Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845875
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

why are you eliminating the hadoop-2.5 profile?


---


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread serranom
Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845855
  
--- Diff: pom.xml ---
@@ -171,7 +171,7 @@
 4.11
 3.2
 1.5
-[2.0.2-alpha,2.3.0]
+2.7.2
 target/hadoop20-classes
--- End diff --

is there any reason to keep this property at all?  seems better to just 
have it defined in each profile as necessary.


---


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread serranom
Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845936
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

this should be protected not private


---


[GitHub] twill pull request #65: (TWILL-254) Update to use ContainerId.fromString

2018-02-04 Thread serranom
Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845929
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

why remove final?


---