[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954090#comment-15954090
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/51#discussion_r109512583
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/CustomClassLoader.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import org.apache.twill.api.ServiceAnnouncer;
+import org.apache.twill.common.Cancellable;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+import org.objectweb.asm.commons.GeneratorAdapter;
+import org.objectweb.asm.commons.Method;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * ClassLoader that generates a new class for the {@link 
CustomClassLoaderTestRun}.
+ */
+public final class CustomClassLoader extends URLClassLoader {
+
+  public CustomClassLoader(URL[] urls, ClassLoader parent) {
+super(urls, parent);
+  }
+
+  @Override
+  protected Class findClass(String name) throws ClassNotFoundException {
+if (!CustomClassLoaderRunnable.GENERATED_CLASS_NAME.equals(name)) {
+  return super.findClass(name);
+}
+
+// Generate a class that look like this:
+//
+// public class Generated {
+//
+//   public void announce(ServiceAnnouncer announcer, String 
serviceName, int port) {
--- End diff --

actually the comment is correct. The generated method has `void` return 
type. The `announcer.announce` method has `Cancellable` return type, but the 
result gets ignored (hence the `mg.pop()` code in line 79).


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954094#comment-15954094
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/51


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-180) TwillController.onTerminated() should pass the exit status of the app to the callback

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954801#comment-15954801
 ] 

ASF GitHub Bot commented on TWILL-180:
--

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/54
  
LGTM


> TwillController.onTerminated() should pass the exit status of the app to the 
> callback
> -
>
> Key: TWILL-180
> URL: https://issues.apache.org/jira/browse/TWILL-180
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.7.0-incubating
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Listening to this callback, a client can only find out whether an app has 
> terminated, but now how. That makes it hard to distinguish error scenarios: 
> Was the app killed by the user? Did it fail? Or did it successfully complete? 
> There should be an easy way for the client to find out.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-207) Better have the cache name purely based on class hash to encourage greater reuse.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892586#comment-15892586
 ] 

ASF GitHub Bot commented on TWILL-207:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/35
  
+1


> Better have the cache name purely based on class hash to encourage greater 
> reuse.
> -
>
> Key: TWILL-207
> URL: https://issues.apache.org/jira/browse/TWILL-207
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-207) Better have the cache name purely based on class hash to encourage greater reuse.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892553#comment-15892553
 ] 

ASF GitHub Bot commented on TWILL-207:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/35
  
As said in the description, you will see the important changes when 
ignoring white space changes


> Better have the cache name purely based on class hash to encourage greater 
> reuse.
> -
>
> Key: TWILL-207
> URL: https://issues.apache.org/jira/browse/TWILL-207
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-207) Better have the cache name purely based on class hash to encourage greater reuse.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892573#comment-15892573
 ] 

ASF GitHub Bot commented on TWILL-207:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/35
  
Ah, yes, sorry, I thought I added the ignore white spaces when load the 
files changes view.


> Better have the cache name purely based on class hash to encourage greater 
> reuse.
> -
>
> Key: TWILL-207
> URL: https://issues.apache.org/jira/browse/TWILL-207
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-207) Better have the cache name purely based on class hash to encourage greater reuse.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892533#comment-15892533
 ] 

ASF GitHub Bot commented on TWILL-207:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/35
  
This PR contains both style and functional change, could you help tell a 
bit where the important  changes happen?


> Better have the cache name purely based on class hash to encourage greater 
> reuse.
> -
>
> Key: TWILL-207
> URL: https://issues.apache.org/jira/browse/TWILL-207
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-186) ApplicationMaster keeps restarting with NPE in the log.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893834#comment-15893834
 ] 

ASF GitHub Bot commented on TWILL-186:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/34


> ApplicationMaster keeps restarting with NPE in the log.
> ---
>
> Key: TWILL-186
> URL: https://issues.apache.org/jira/browse/TWILL-186
> Project: Apache Twill
>  Issue Type: Bug
>  Components: core, yarn
>Affects Versions: 0.7.0-incubating
>Reporter: Sagar Kapare
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Seems like certain combination of the container sizes launched by AM, causing 
> the AM to keep restarting.
> Following exception is seen in the app master container log:
> {noformat}
> Aug 12, 2016 4:37:39 PM 
> com.google.common.util.concurrent.AbstractExecutionThreadService$1$1 run
> WARNING: Error while attempting to shut down the service after failure.
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.decResourceRequest(AMRMClientImpl.java:687)
> at 
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.removeContainerRequest(AMRMClientImpl.java:477)
> at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:116)
> at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:45)
> at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.allocate(AbstractYarnAMClient.java:119)
> at 
> org.apache.twill.internal.appmaster.ApplicationMasterService.doStop(ApplicationMasterService.java:281)
> at 
> org.apache.twill.internal.AbstractTwillService.shutDown(AbstractTwillService.java:186)
> at 
> com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:55)
> at java.lang.Thread.run(Thread.java:745)
> Exception in thread "ApplicationMasterService" java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.decResourceRequest(AMRMClientImpl.java:687)
> at 
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.removeContainerRequest(AMRMClientImpl.java:477)
> at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:116)
> at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:45)
> at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.allocate(AbstractYarnAMClient.java:119)
> at 
> org.apache.twill.internal.appmaster.ApplicationMasterService.doRun(ApplicationMasterService.java:369)
> at 
> org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:179)
> at 
> com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-176) Set ResourceManager tracking URL to resources path.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954257#comment-15954257
 ] 

ASF GitHub Bot commented on TWILL-176:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/52


> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-176) Set ResourceManager tracking URL to resources path.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954214#comment-15954214
 ] 

ASF GitHub Bot commented on TWILL-176:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/52

(TWILL-176) Redirect all requests to tracker URL to /resources path

- "/resources" is the only path supported for the tracker service for now

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-176

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/52.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #52


commit b07fd47e3e78ad80097e26a944baceebefd0559c
Author: Terence Yim 
Date:   2017-04-03T21:42:44Z

(TWILL-176) Redirect all requests to tracker URL to /resources path

- "/resources" is the only path supported for the tracker service for now




> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954077#comment-15954077
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/51#discussion_r109510575
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/CustomClassLoader.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import org.apache.twill.api.ServiceAnnouncer;
+import org.apache.twill.common.Cancellable;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+import org.objectweb.asm.commons.GeneratorAdapter;
+import org.objectweb.asm.commons.Method;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * ClassLoader that generates a new class for the {@link 
CustomClassLoaderTestRun}.
+ */
+public final class CustomClassLoader extends URLClassLoader {
+
+  public CustomClassLoader(URL[] urls, ClassLoader parent) {
+super(urls, parent);
+  }
+
+  @Override
+  protected Class findClass(String name) throws ClassNotFoundException {
+if (!CustomClassLoaderRunnable.GENERATED_CLASS_NAME.equals(name)) {
+  return super.findClass(name);
+}
+
+// Generate a class that look like this:
+//
+// public class Generated {
+//
+//   public void announce(ServiceAnnouncer announcer, String 
serviceName, int port) {
--- End diff --

Will update.


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107620#comment-16107620
 ] 

ASF GitHub Bot commented on TWILL-147:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/57
  
One quick question, does Kafka 0.10 client works with older version of 
Kafka server?

Also, ideally we should separate out all these Kafka related dependencies 
in a separate twill module, otherwise making it compatible with different Scala 
version would be quite difficult.


> Allow using external Kafka server for log collection
> 
>
> Key: TWILL-147
> URL: https://issues.apache.org/jira/browse/TWILL-147
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, core
>Reporter: Terence Yim
>Assignee: Filippov Sergey
>
> Show allow user to use any Kafka server for log collection, not necessarily 
> starting an embedded one in the AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107645#comment-16107645
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r130407739
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -205,8 +208,9 @@ private RunningContainers 
createRunningContainers(ContainerId appMasterContainer
   Integer.parseInt(System.getenv(EnvKeys.YARN_CONTAINER_MEMORY_MB)),
   appMasterHost, null);
 String appId = 
appMasterContainerId.getApplicationAttemptId().getApplicationId().toString();
-return new RunningContainers(appId, appMasterResources, zkClient, 
applicationLocation,
-  twillSpec.getRunnables(), twillRuntimeSpec.getMaxRetries());
+return new RunningContainers(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId(),
--- End diff --

Better just pass the `twillRuntimeSpec` to the `RunningContainers` instead 
of getting individual entries one by one. It can replace the 
`twillRuntimeSpec.getXXX` and `twillSpec.getRunnables`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107647#comment-16107647
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r130406951
  
--- Diff: twill-api/src/main/java/org/apache/twill/api/EventHandler.java ---
@@ -124,6 +124,74 @@ public void initialize(EventHandlerContext context) {
   }
 
   /**
+   * Invoked by the application when it starts.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   */
+  public void started(String twillAppName, RunId runId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when new container is launched for {@link 
TwillRunnable}.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   * @param runnableName name of the runnable to be run in the new 
container
+   * @param instanceId the instance ID of the runnable instance to be run 
in the new container
+   * @param containerId the ID of the newly launched container
+   */
+  public void containerLaunched(String twillAppName, RunId runId, String 
runnableName,
+int instanceId, String containerId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when a container is stopped.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   * @param runnableName name of the runnable in the stopped container
+   * @param instanceId the instance ID of the runnable instance run in the 
stopped container
+   * @param containerId the ID of the stopped container
+   */
+  public void containerStopped(String twillAppName, RunId runId, String 
runnableName,
--- End diff --

Should provide the exit code as well.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113163#comment-16113163
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131210382
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -291,6 +301,23 @@ public void run() {
 // Since all the runnables are now stopped, it is okay to stop the 
poller.
 stopPoller.shutdownNow();
 cleanupDir();
+if (eventHandler != null) {
+  if (finalStatus == null) {
+// if finalStatus is not set, the application must be stopped by a 
SystemMessages#STOP_COMMAND
+eventHandler.killed(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId());
+  }
+  switch (finalStatus) {
--- End diff --

This may have NPE. Should have a `else`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113159#comment-16113159
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131208560
  
--- Diff: twill-api/src/main/java/org/apache/twill/api/EventHandler.java ---
@@ -124,6 +124,74 @@ public void initialize(EventHandlerContext context) {
   }
 
   /**
+   * Invoked by the application when it starts.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   */
+  public void started(String twillAppName, RunId runId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when new container is launched for {@link 
TwillRunnable}.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   * @param runnableName name of the runnable to be run in the new 
container
+   * @param instanceId the instance ID of the runnable instance to be run 
in the new container
+   * @param containerId the ID of the newly launched container
+   */
+  public void containerLaunched(String twillAppName, RunId runId, String 
runnableName,
+int instanceId, String containerId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when a container is stopped.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   * @param runnableName name of the runnable in the stopped container
+   * @param instanceId the instance ID of the runnable instance run in the 
stopped container
+   * @param containerId the ID of the stopped container
+   */
+  public void containerStopped(String twillAppName, RunId runId, String 
runnableName,
+   int instanceId, String containerId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when all containers complete.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   */
+  public void completed(String twillAppName, RunId runId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when stop command is received to kill the 
current application.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   */
+  public void killed(String twillAppName, RunId runId) {
+// No-op
+  }
+
+  /**
+   * Invoked by the application when the application is aborted because of 
timeout.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   */
+  public void aborted(String twillAppName, RunId runId) {
+// No-op
+  }
+
+  /**
* Invoked by the application when shutting down.
*/
   public void destroy() {
--- End diff --

continue, with the recheck time to some hardcode constant (say 60 seconds).


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113165#comment-16113165
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131212239
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/ProvisionTimeoutTestRun.java ---
@@ -79,12 +83,22 @@ public void initialize(EventHandlerContext context) {
 
 @Override
 public TimeoutAction launchTimeout(Iterable 
timeoutEvents) {
+  for (TimeoutEvent event : timeoutEvents) {
+LOG.info("Requested {} containers for runnable {}, only got {} 
after {} ms.",
+ event.getExpectedInstances(), event.getRunnableName(),
+ event.getActualInstances(), System.currentTimeMillis() - 
event.getRequestTime());
+  }
   if (abort) {
 return TimeoutAction.abort();
   } else {
 return TimeoutAction.recheck(10, TimeUnit.SECONDS);
   }
 }
+
+@Override
+public void aborted(String twillAppName, RunId runId) {
+  LOG.info(String.format("Aborted %s with runId %s", twillAppName, 
runId.getId()));
--- End diff --

Besides logging, can you add/modify the test to actually validate the new 
lifecycle methods in the EventHandler?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113164#comment-16113164
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131209163
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -135,6 +135,15 @@
   private final Map> environments;
   private final TwillRuntimeSpecification twillRuntimeSpec;
 
+  /**
+   * Final status of this service when it stops.
+   */
+  public enum FinalStatus {
--- End diff --

Sounds more like `StopStatus` or `CompletionStatus`. Also it should be 
`private` since it is only used in this class.

Also, please move it before all fields declaration instead of having it in 
between fields.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113161#comment-16113161
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131210237
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -291,6 +301,23 @@ public void run() {
 // Since all the runnables are now stopped, it is okay to stop the 
poller.
 stopPoller.shutdownNow();
 cleanupDir();
+if (eventHandler != null) {
+  if (finalStatus == null) {
+// if finalStatus is not set, the application must be stopped by a 
SystemMessages#STOP_COMMAND
+eventHandler.killed(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId());
+  }
+  switch (finalStatus) {
+case COMPLETED:
+  eventHandler.completed(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId());
+  break;
+case ABORTED:
+eventHandler.aborted(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId());
+  break;
+default:
+  // should never reach here
+  LOG.error("Unsupported FinalStatus '%s'", finalStatus.name());
--- End diff --

That's not the right syntax. slf4j logger uses `{}`. Should be 
`LOG.error("Unsupported status {}", finalStatus)`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-237) Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113243#comment-16113243
 ] 

ASF GitHub Bot commented on TWILL-237:
--

Github user yufeldman commented on a diff in the pull request:

https://github.com/apache/twill/pull/55#discussion_r131224027
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/filesystem/FileContextLocationUtil.java
 ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.filesystem;
+
+import com.google.common.base.Throwables;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.HAUtil;
+
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.net.URI;
+
+/**
+ * Utility class.
+ */
+final class FileContextLocationUtil {
--- End diff --

Thank you for the final review. Addressed your comment


> Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8
> --
>
> Key: TWILL-237
> URL: https://issues.apache.org/jira/browse/TWILL-237
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>
> Twill is using hdfs.HAUtil apis that are suppose to be hdfs private and 
> subsequently signature of isLogicalURI was changed (actually name was 
> changed) in hadoop version 2.8
> Will post a patch for now to support both old and new names, but I think 
> eventually references to private hdfs interfaces/classes should be removed 
> from twill



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113254#comment-16113254
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131225251
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/ProvisionTimeoutTestRun.java ---
@@ -79,12 +83,22 @@ public void initialize(EventHandlerContext context) {
 
 @Override
 public TimeoutAction launchTimeout(Iterable 
timeoutEvents) {
+  for (TimeoutEvent event : timeoutEvents) {
+LOG.info("Requested {} containers for runnable {}, only got {} 
after {} ms.",
--- End diff --

i was using it to confirm the change. Will remove


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113320#comment-16113320
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131234842
  
--- Diff: twill-api/src/main/java/org/apache/twill/api/EventHandler.java ---
@@ -124,6 +124,75 @@ public void initialize(EventHandlerContext context) {
   }
 
   /**
+   * Invoked by the application when it starts.
+   *
+   * @param twillAppName name of the current application
+   * @param runId run ID of current application run
+   */
+  public void started(String twillAppName, RunId runId) {
--- End diff --

the `EventHandlerContext` just contains a `EventHandlerSpecification`, and 
it doesn't seem to guarantee app name and runid are there?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-238) Change TwillController restartInstances API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087024#comment-16087024
 ] 

ASF GitHub Bot commented on TWILL-238:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/56#discussion_r126296669
  
--- Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java 
---
@@ -17,15 +17,14 @@
  */
 package org.apache.twill.api;
 
-import org.apache.twill.api.logging.LogEntry;
-import org.apache.twill.api.logging.LogHandler;
-import org.apache.twill.discovery.Discoverable;
-import org.apache.twill.discovery.ServiceDiscovered;
-
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.Future;
 import javax.annotation.Nullable;
+import org.apache.twill.api.logging.LogEntry;
--- End diff --

Please don't rearrange the imports. You can use the ide profiles available 
in http://twill.apache.org/HowToContribute.html


> Change TwillController restartInstances API
> ---
>
> Key: TWILL-238
> URL: https://issues.apache.org/jira/browse/TWILL-238
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Filippov Sergey
>Assignee: Filippov Sergey
>Priority: Minor
> Fix For: 0.12.0
>
>
> Lets change signature from Future restartInstances(String runnable, 
> int instanceId, int... moreInstanceIds) to Future 
> restartInstances(String runnable, Interger instanceId, Integer... 
> moreInstanceIds) so it will be easier to convert from collections to vararg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-238) Change TwillController restartInstances API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079099#comment-16079099
 ] 

ASF GitHub Bot commented on TWILL-238:
--

GitHub user fonelor opened a pull request:

https://github.com/apache/twill/pull/56

implemented issue TWILL-238: restart instances with Set of Integers

Implemented https://issues.apache.org/jira/browse/TWILL-238 

New method in TwillController to restart instances of runnable

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fonelor/twill feature/238_restartinstances_api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #56


commit eb184e6aa727d93fce4e6e4e33c729eb09017750
Author: Sergey Filippov 
Date:   2017-07-08T11:24:22Z

implemented issue TWILL-238: restart instances with Set of Integers




> Change TwillController restartInstances API
> ---
>
> Key: TWILL-238
> URL: https://issues.apache.org/jira/browse/TWILL-238
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Filippov Sergey
>Priority: Minor
>
> Lets change signature from Future restartInstances(String runnable, 
> int instanceId, int... moreInstanceIds) to Future 
> restartInstances(String runnable, Interger instanceId, Integer... 
> moreInstanceIds) so it will be easier to convert from collections to vararg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-238) Change TwillController restartInstances API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087129#comment-16087129
 ] 

ASF GitHub Bot commented on TWILL-238:
--

Github user fonelor commented on the issue:

https://github.com/apache/twill/pull/56
  
Reverted reformatting


> Change TwillController restartInstances API
> ---
>
> Key: TWILL-238
> URL: https://issues.apache.org/jira/browse/TWILL-238
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Filippov Sergey
>Assignee: Filippov Sergey
>Priority: Minor
> Fix For: 0.12.0
>
>
> Lets change signature from Future restartInstances(String runnable, 
> int instanceId, int... moreInstanceIds) to Future 
> restartInstances(String runnable, Interger instanceId, Integer... 
> moreInstanceIds) so it will be easier to convert from collections to vararg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-238) Change TwillController restartInstances API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087128#comment-16087128
 ] 

ASF GitHub Bot commented on TWILL-238:
--

Github user fonelor commented on a diff in the pull request:

https://github.com/apache/twill/pull/56#discussion_r127425262
  
--- Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java 
---
@@ -17,15 +17,14 @@
  */
 package org.apache.twill.api;
 
-import org.apache.twill.api.logging.LogEntry;
-import org.apache.twill.api.logging.LogHandler;
-import org.apache.twill.discovery.Discoverable;
-import org.apache.twill.discovery.ServiceDiscovered;
-
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.Future;
 import javax.annotation.Nullable;
+import org.apache.twill.api.logging.LogEntry;
--- End diff --

i've reverted reformatting changes


> Change TwillController restartInstances API
> ---
>
> Key: TWILL-238
> URL: https://issues.apache.org/jira/browse/TWILL-238
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Filippov Sergey
>Assignee: Filippov Sergey
>Priority: Minor
> Fix For: 0.12.0
>
>
> Lets change signature from Future restartInstances(String runnable, 
> int instanceId, int... moreInstanceIds) to Future 
> restartInstances(String runnable, Interger instanceId, Integer... 
> moreInstanceIds) so it will be easier to convert from collections to vararg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-238) Change TwillController restartInstances API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089212#comment-16089212
 ] 

ASF GitHub Bot commented on TWILL-238:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/56


> Change TwillController restartInstances API
> ---
>
> Key: TWILL-238
> URL: https://issues.apache.org/jira/browse/TWILL-238
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Filippov Sergey
>Assignee: Filippov Sergey
>Priority: Minor
> Fix For: 0.12.0
>
>
> Lets change signature from Future restartInstances(String runnable, 
> int instanceId, int... moreInstanceIds) to Future 
> restartInstances(String runnable, Interger instanceId, Integer... 
> moreInstanceIds) so it will be easier to convert from collections to vararg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096014#comment-16096014
 ] 

ASF GitHub Bot commented on TWILL-147:
--

GitHub user fonelor opened a pull request:

https://github.com/apache/twill/pull/57

(TWILL-147) Allow using external Kafka server for log collection

Now external kafka servers could be used for log aggregation
Kafka updated to 0.10.2.0
Embedded kafka server starts if log aggregation is enabled and no external 
kafka servers configured
removed old zk based kafka broker discovery service

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fonelor/twill feature/kafka_upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/57.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #57


commit 6816ea9945d76921363960043b0bfc392d320cd9
Author: Sergey Filippov 
Date:   2017-07-14T09:47:21Z

started kafka migration

commit 9f94b4e38ff7bcd8356f9c5671537c9787f93eb6
Author: Sergey Filippov 
Date:   2017-07-18T15:51:47Z

implemented new KafkaConsumer and new KafkaProducer

commit bd4803e8959b632f9c09ddfc4ec644a461175eea
Author: Sergey Filippov 
Date:   2017-07-19T15:20:03Z

fixed jmx problems with client.id for consumer
fixed no payload in logger FetchedMessage
fixed incorrect scheduling and running of KafkaConsumer task

commit e98d2f4f92d15a115ef260095ab2f95b6b990417
Author: Sergey Filippov 
Date:   2017-07-20T15:33:20Z

fixed KafkaTest
removed Encoders
implemented broker.id generation

commit 8bbcb1a417fbeaa22d51b7307c13e1efa3ade71d
Author: Sergey Filippov 
Date:   2017-07-21T08:37:23Z

checkStyle

commit 92937a5b76c368af7ee86f2a495670aaa4d8c6bb
Author: Sergey Filippov 
Date:   2017-07-21T08:50:31Z

fixed compressing setting

commit 71d3db756a79f897c4265443a400375ec9f386bb
Author: Sergey Filippov 
Date:   2017-07-21T09:06:07Z

clean up
revert code moving
added slf4j-log4j12 for testing with kafka

commit 7667767cd54981f55bee12c3cecc69cc5bd00cbd
Author: Sergey Filippov 
Date:   2017-07-21T09:15:20Z

added kafka bootstrap servers configuration to TwillPreparer

commit cbc7a554e43b80a6622f1a065a3501dc27d88e64
Author: Sergey Filippov 
Date:   2017-07-21T09:18:59Z

Removed old kafka services




> Allow using external Kafka server for log collection
> 
>
> Key: TWILL-147
> URL: https://issues.apache.org/jira/browse/TWILL-147
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, core
>Reporter: Terence Yim
>Assignee: Filippov Sergey
>
> Show allow user to use any Kafka server for log collection, not necessarily 
> starting an embedded one in the AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101281#comment-16101281
 ] 

ASF GitHub Bot commented on TWILL-240:
--

GitHub user maochf opened a pull request:

https://github.com/apache/twill/pull/58

[TWILL-240] EventHandler Improvement

https://issues.apache.org/jira/browse/TWILL-240

Add started, containerLaunched, containerStopped, completed, killed, 
aborted to EventHandler

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maochf/twill feature/event-handler-improvement

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/58.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #58


commit 76eeca04586712123c44caf5788c17e0968a
Author: Chengfeng 
Date:   2017-07-25T19:53:26Z

[TWILL-240] Add started, containerLaunched, containerStopped, completed, 
killed, aborted to EventHandler




> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102010#comment-16102010
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r129647384
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -222,6 +226,7 @@ protected void doStart() throws Exception {
 // initialize the event handler, if it fails, it will fail the 
application.
 if (eventHandler != null) {
   eventHandler.initialize(new 
BasicEventHandlerContext(twillSpec.getEventHandler()));
+  eventHandler.started(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId());
--- End diff --

If initialization fails, then the whole app will fail. `TwillController` 
should be able to catch this


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101983#comment-16101983
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user sameetandpotatoes commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r129643121
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +281,9 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  eventHandler.containerLaunched(twillAppName, runId, 
runnableName, instanceId, containerId);
--- End diff --

Typo - should be `containerStopped`


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101982#comment-16101982
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user sameetandpotatoes commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r129643180
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -103,6 +105,8 @@ public Integer apply(BitSet input) {
   private final Table containers;
 
   // Map from runnableName to a BitSet, with the  bit turned 
on for having an instance running.
+  private final String twillAppName;
--- End diff --

The comment above this line should be moved down to above the 
`runnableInstances` variable


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-237) Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114666#comment-16114666
 ] 

ASF GitHub Bot commented on TWILL-237:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/55


> Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8
> --
>
> Key: TWILL-237
> URL: https://issues.apache.org/jira/browse/TWILL-237
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
> Fix For: 0.12.0
>
>
> Twill is using hdfs.HAUtil apis that are suppose to be hdfs private and 
> subsequently signature of isLogicalURI was changed (actually name was 
> changed) in hadoop version 2.8
> Will post a patch for now to support both old and new names, but I think 
> eventually references to private hdfs interfaces/classes should be removed 
> from twill



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115179#comment-16115179
 ] 

ASF GitHub Bot commented on TWILL-241:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/59

(TWILL-241) Added per runnable configuration and jvm options support

This PR has two commits, one for adding per runnable configuration, the 
other for adding per runnable jvm options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-241-per-runnable-opts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/59.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #59


commit f1931deb128dda8b3c4da76486ab1b443318496e
Author: Terence Yim 
Date:   2017-08-04T22:29:05Z

(TWILL-241) Added support for per Runnable configuration

commit 29a7999f45859996595287fb1c28b225a2564ed9
Author: Terence Yim 
Date:   2017-08-04T23:19:32Z

(TWILL-241) Added support for per runnable JVM options
- Also removed JvmOptionsCodec since JvmOptions only uses simple types




> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115184#comment-16115184
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/59#discussion_r131511104
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/ContainerSizeTestRun.java ---
@@ -67,20 +69,37 @@ public void testContainerSize() throws 
InterruptedException, TimeoutException, E
   @Test
   public void testMaxHeapSize() throws InterruptedException, 
TimeoutException, ExecutionException {
 TwillRunner runner = getTwillRunner();
+String runnableName = "sleep";
+
 TwillController controller = runner.prepare(new MaxHeapApp())
-  // Alter the AM container size
-  
.withConfiguration(Collections.singletonMap(Configs.Keys.YARN_AM_MEMORY_MB, 
"256"))
+  // Alter the AM container size and heap ratio
+  .withConfiguration(ImmutableMap.of(Configs.Keys.YARN_AM_MEMORY_MB, 
"256",
+ 
Configs.Keys.HEAP_RESERVED_MIN_RATIO, "0.65"))
+  // Use a different heap ratio and reserved memory size for the 
runnable
+  .withConfiguration(runnableName,
+ 
ImmutableMap.of(Configs.Keys.HEAP_RESERVED_MIN_RATIO, "0.8",
--- End diff --

can you add a test that, if this config for the runnable does not have 
HEAP_RESERVED_MIN_RATIO, it defaults to the one above (0.65) and not the system 
default.  



> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115192#comment-16115192
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/59#discussion_r131511384
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -183,6 +184,13 @@ public TwillPreparer withConfiguration(Map config) {
   }
 
   @Override
+  public TwillPreparer withConfiguration(String runnableName, Map config) {
--- End diff --

Should default to the app setting first, then to the constants default


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115190#comment-16115190
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/59#discussion_r131511349
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java
 ---
@@ -87,19 +86,46 @@ public String getTwillAppName() {
 return twillAppName;
   }
 
-  public int getReservedMemory() {
-return reservedMemory;
+  /**
+   * Returns the minimum heap ratio for the application master.
+   */
+  public double getAMMinHeapRatio() {
+return getMinHeapRatio(config);
+  }
+
+  /**
+   * Returns the minimum heap ratio for the given runnable.
+   */
+  public double getMinHeapRatio(String runnableName) {
+return getMinHeapRatio(runnableConfigs.containsKey(runnableName) ? 
runnableConfigs.get(runnableName) : config);
--- End diff --

Agree. Get it wrong after some refactoring


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115180#comment-16115180
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/59#discussion_r131510954
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -183,6 +184,13 @@ public TwillPreparer withConfiguration(Map config) {
   }
 
   @Override
+  public TwillPreparer withConfiguration(String runnableName, Map config) {
--- End diff --

If the config contains only one key, say reserved memory, will the 
remaining keys default to the values configured for the entire app? Or will 
they default to the system default?


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119072#comment-16119072
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132037568
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -221,7 +229,8 @@ protected void doStart() throws Exception {
 
 // initialize the event handler, if it fails, it will fail the 
application.
 if (eventHandler != null) {
-  eventHandler.initialize(new 
BasicEventHandlerContext(twillSpec.getEventHandler()));
+  eventHandler.initialize(new 
BasicEventHandlerContext(twillRuntimeSpec, twillSpec.getEventHandler()));
--- End diff --

The `twillRuntimeSpec` already contains `twillSpec`. Seems unnecessary to 
pass in `twillSpec.getEventHandler()` as a separate parameter.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119073#comment-16119073
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132038521
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +280,14 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  Integer exitStatus = containerExitStatus.get(containerId);
--- End diff --

Should remove it from the exit status map, otherwise the map can keep 
growing in size.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119074#comment-16119074
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132040093
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +280,14 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  Integer exitStatus = containerExitStatus.get(containerId);
+  if (exitStatus == null) {
--- End diff --

So this is for handling the case when `handleCompleted` was not called 
(e.g. the runnable container is not stopping after receiving the "stop" 
command), right? If that's the case, isn't that the exit status is always not 
there?

Also, do we need to do the similar logic to call event handler in the 
`stopAll` method?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119075#comment-16119075
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132038164
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -173,6 +183,9 @@ void start(String runnableName, ContainerInfo 
containerInfo, TwillContainerLaunc
 startSequence.addLast(runnableName);
   }
   containerChange.signalAll();
+  if (eventHandler != null) {
+eventHandler.containerLaunched(runnableName, instanceId, 
containerInfo.getId());
--- End diff --

How do we handle exception raised from event handler methods?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119110#comment-16119110
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132048851
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -267,6 +280,14 @@ void stopByIdAndWait(String runnableName, int 
instanceId) {
 
 resourceReport.removeRunnableResources(runnableName, containerId);
 containerChange.signalAll();
+if (eventHandler != null) {
+  Integer exitStatus = containerExitStatus.get(containerId);
+  if (exitStatus == null) {
--- End diff --

`controller.stopAndWait();` at line 269 blocks until `handleCompleted` 
method runs


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119022#comment-16119022
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132029465
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/EventHandlerTest.java ---
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableMap;
+import org.apache.twill.api.AbstractTwillRunnable;
+import org.apache.twill.api.EventHandler;
+import org.apache.twill.api.EventHandlerContext;
+import org.apache.twill.api.TwillApplication;
+import org.apache.twill.api.TwillController;
+import org.apache.twill.api.TwillSpecification;
+import org.apache.twill.api.logging.PrinterLogHandler;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * Tests {@link EventHandler} methods
+ */
+public final class EventHandlerTest extends BaseYarnTest {
+  private static final Logger LOG = 
LoggerFactory.getLogger(EventHandlerTest.class);
+
+  @ClassRule
+  public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
+  public static final String STARTED_FILE = "started_file";
+  public static final String RUN_FILE = "run_file";
+  public static final String CONTAINER_LAUNCHED_FOLDER = "launched_folder";
+  public static final String CONTAINER_STOPPED_FOLDER = "stopped_folder";
+  public static final String COMPLETED_FILE = "completed_file";
+  public static final String KILLED_FILE = "killed_file";
+  public static final String ABORTED_FILE = "aborted_file";
+
+  @Test
+  public void testComplete() throws InterruptedException, 
ExecutionException, TimeoutException, IOException {
+// Create a parent folder to be written by EventHandler
+File parentFolder = TMP_FOLDER.newFolder();
+parentFolder.setWritable(true, false);
+TwillController controller = getTwillRunner().prepare(new 
CompleteApplication(parentFolder.getAbsolutePath()))
+  .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out, 
true)))
+  .start();
+
+// Wait for the app to complete within 120 seconds.
+try {
+  controller.awaitTerminated(120, TimeUnit.SECONDS);
+  Set handlerFiles = new 
HashSet<>(Arrays.asList(parentFolder.list()));
+  Assert.assertEquals(5, handlerFiles.size());
+  // EventHandler#started() method should be called to create a file
+  Assert.assertTrue(handlerFiles.contains(STARTED_FILE));
+  // CompleteRunnable#run() method should be called to create a file 
after EventHandler#started() method is called
+  Assert.assertTrue(handlerFiles.contains(RUN_FILE));
+  // EventHandler#containerLaunched(String, int, String) method should 
be called to create a folder
+  Assert.assertTrue(handlerFiles.contains(CONTAINER_LAUNCHED_FOLDER));
+  // EventHandler#containerStopped(String, int, String, int) method 
should be called to create a folder
+  Assert.assertTrue(handlerFiles.contains(CONTAINER_STOPPED_FOLDER));
+  // Assert that containerLaunched and containerStopped are called for 
the 

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119021#comment-16119021
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132029365
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/EventHandlerTest.java ---
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableMap;
+import org.apache.twill.api.AbstractTwillRunnable;
+import org.apache.twill.api.EventHandler;
+import org.apache.twill.api.EventHandlerContext;
+import org.apache.twill.api.TwillApplication;
+import org.apache.twill.api.TwillController;
+import org.apache.twill.api.TwillSpecification;
+import org.apache.twill.api.logging.PrinterLogHandler;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * Tests {@link EventHandler} methods
+ */
+public final class EventHandlerTest extends BaseYarnTest {
--- End diff --

Rename the test class to `EventHandlerTestRun` and add it to 
`YarnTestSuite`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120612#comment-16120612
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132297954
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java
 ---
@@ -401,6 +409,24 @@ void stopAll() {
 // When we acquire this lock, all stopped runnables should have been 
cleaned up by handleCompleted() method
 containerLock.lock();
 try {
+  for (Map.Entry> entry 
: containers.rowMap().entrySet()) {
+String runnableName = entry.getKey();
+Collection containerInfos = 
containerStats.get(runnableName);
+for (Map.Entry 
containerControllerEntry : entry.getValue().entrySet()) {
+  boolean containerExist = false;
+  for (ContainerInfo containerInfo : containerInfos) {
+if 
(containerInfo.getId().equals(containerControllerEntry.getKey())) {
+  containerExist = true;
+  break;
+}
+  }
+  // Only call eventHandler.containerStopped if container is not 
removed by handleCompleted
+  if (containerExist) {
+eventHandler.containerStopped(runnableName, 
containerControllerEntry.getValue().getInstanceId(),
--- End diff --

you can move this inside the `for` loop and no need to use `containerExist`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120602#comment-16120602
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132295847
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -181,7 +189,6 @@ public Reader getInput() throws IOException {
   }
 
   @SuppressWarnings("unchecked")
-  @Nullable
   private EventHandler createEventHandler(TwillSpecification twillSpec) 
throws ClassNotFoundException {
 // Should be able to load by this class ClassLoader, as they packaged 
in the same jar.
 EventHandlerSpecification handlerSpec = twillSpec.getEventHandler();
--- End diff --

if `handlerSpec` is `null`, you still want to return a no-op handler, right?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120603#comment-16120603
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132296198
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -193,7 +200,91 @@ private EventHandler 
createEventHandler(TwillSpecification twillSpec) throws Cla
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+if (delegate == null) {
--- End diff --

`Instances.newInstance` never return `null`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120614#comment-16120614
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132298585
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -193,7 +200,91 @@ private EventHandler 
createEventHandler(TwillSpecification twillSpec) throws Cla
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+if (delegate == null) {
+  // if no handler is specified, return an EventHandler with no-op
+  return new EventHandler() {};
+}
+// wrap the delegate EventHandler so that all errors will be caught
+return new EventHandler() {
--- End diff --

originally, if `EventHandler#initialize` fails, the app will fail to start. 
Do we want to change this behavior?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120609#comment-16120609
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132297305
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -193,7 +200,91 @@ private EventHandler 
createEventHandler(TwillSpecification twillSpec) throws Cla
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+if (delegate == null) {
+  // if no handler is specified, return an EventHandler with no-op
+  return new EventHandler() {};
+}
+// wrap the delegate EventHandler so that all errors will be caught
+return new EventHandler() {
--- End diff --

You'll need to wrap the `initialize` as well. Also, I'd suggest if 
`initialize` throw exception, you'll just ignore all calls to other lifecycle 
methods to avoid excessive logging.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120746#comment-16120746
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322804
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -239,14 +359,8 @@ protected void doStop() throws Exception {
 LOG.info("Stop application master with spec: {}",
  
TwillRuntimeSpecificationAdapter.create().toJson(twillRuntimeSpec));
 
-if (eventHandler != null) {
-  try {
-// call event handler destroy. If there is error, only log and not 
affected stop sequence.
-eventHandler.destroy();
-  } catch (Throwable t) {
-LOG.warn("Exception when calling {}.destroy()", 
eventHandler.getClass().getName(), t);
-  }
-}
+// call event handler destroy
+eventHandler.destroy();
--- End diff --

call it at the end of `doStop() `?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-242) Extra JVM options is dropped when custom classloader is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120717#comment-16120717
 ] 

ASF GitHub Bot commented on TWILL-242:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/60

(TWILL-242) Fixed the dropping of extra jvm opts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-242-fix-extra-jvm-opts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/60.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #60


commit 7187ed966139fd05ce4a6a8d79b65feac34545ae
Author: Terence Yim 
Date:   2017-08-09T21:56:33Z

(TWILL-242) Fixed the dropping of extra jvm opts




> Extra JVM options is dropped when custom classloader is used
> 
>
> Key: TWILL-242
> URL: https://issues.apache.org/jira/browse/TWILL-242
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> It is introduced by the fix in TWILL-241



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120745#comment-16120745
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322316
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -239,14 +359,8 @@ protected void doStop() throws Exception {
 LOG.info("Stop application master with spec: {}",
  
TwillRuntimeSpecificationAdapter.create().toJson(twillRuntimeSpec));
 
-if (eventHandler != null) {
-  try {
-// call event handler destroy. If there is error, only log and not 
affected stop sequence.
-eventHandler.destroy();
-  } catch (Throwable t) {
-LOG.warn("Exception when calling {}.destroy()", 
eventHandler.getClass().getName(), t);
-  }
-}
+// call event handler destroy
+eventHandler.destroy();
--- End diff --

Shouldn't call `destroy()` here anymore. It should be called after call 
other calls to the event handler, as it is acted as a cleanup call.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120748#comment-16120748
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322898
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -239,14 +359,8 @@ protected void doStop() throws Exception {
 LOG.info("Stop application master with spec: {}",
  
TwillRuntimeSpecificationAdapter.create().toJson(twillRuntimeSpec));
 
-if (eventHandler != null) {
-  try {
-// call event handler destroy. If there is error, only log and not 
affected stop sequence.
-eventHandler.destroy();
-  } catch (Throwable t) {
-LOG.warn("Exception when calling {}.destroy()", 
eventHandler.getClass().getName(), t);
-  }
-}
+// call event handler destroy
+eventHandler.destroy();
--- End diff --

I think so.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120743#comment-16120743
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r132322087
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -181,19 +190,130 @@ public Reader getInput() throws IOException {
   }
 
   @SuppressWarnings("unchecked")
-  @Nullable
   private EventHandler createEventHandler(TwillSpecification twillSpec) 
throws ClassNotFoundException {
 // Should be able to load by this class ClassLoader, as they packaged 
in the same jar.
 EventHandlerSpecification handlerSpec = twillSpec.getEventHandler();
 if (handlerSpec == null) {
-  return null;
+  // if no handler is specified, return an EventHandler with no-op
+  return new EventHandler() {};
 }
 
 Class handlerClass = 
getClass().getClassLoader().loadClass(handlerSpec.getClassName());
 
Preconditions.checkArgument(EventHandler.class.isAssignableFrom(handlerClass),
 "Class {} does not implements {}",
 handlerClass, 
EventHandler.class.getName());
-return Instances.newInstance((Class) 
handlerClass);
+final EventHandler delegate = Instances.newInstance((Class) handlerClass);
+// wrap all calls to the delegate EventHandler methods except 
initialize so that all errors will be caught
+return new EventHandler() {
+  private boolean initialized;
+
+  @Override
+  public void initialize(EventHandlerContext context) {
+delegate.initialize(context);
+initialized = true;
--- End diff --

Since any exception thrown from `initialize` will terminate the app, I 
think we don't need this, right?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-242) Extra JVM options is dropped when custom classloader is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120799#comment-16120799
 ] 

ASF GitHub Bot commented on TWILL-242:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/60


> Extra JVM options is dropped when custom classloader is used
> 
>
> Key: TWILL-242
> URL: https://issues.apache.org/jira/browse/TWILL-242
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> It is introduced by the fix in TWILL-241



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120825#comment-16120825
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user maochf commented on the issue:

https://github.com/apache/twill/pull/58
  
squashed


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120944#comment-16120944
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/58


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117271#comment-16117271
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131761843
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
--- End diff --

Also please add javadoc.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117277#comment-16117277
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131762365
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
+
+  String getTwillAppRunId();
--- End diff --

The run id should be of type `RunId`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117272#comment-16117272
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131761743
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
--- End diff --

Should be called `getApplicationName()`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117276#comment-16117276
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131761787
  
--- Diff: 
twill-api/src/main/java/org/apache/twill/api/EventHandlerContext.java ---
@@ -22,5 +22,9 @@
  */
 public interface EventHandlerContext {
 
+  String getTwillAppName();
+
+  String getTwillAppRunId();
--- End diff --

Just call it `getRunId()`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117383#comment-16117383
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user anew commented on the issue:

https://github.com/apache/twill/pull/59
  
LGTM


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117274#comment-16117274
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131762894
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/ProvisionTimeoutTestRun.java ---
@@ -85,13 +105,29 @@ public TimeoutAction 
launchTimeout(Iterable timeoutEvents) {
 return TimeoutAction.recheck(10, TimeUnit.SECONDS);
   }
 }
+
+@Override
+public void aborted() {
+  try {
+new 
File(context.getSpecification().getConfigs().get("parentFolderPath") + 
File.separator
--- End diff --

Also, we are only testing abort? How about other lifecycle methods?


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117273#comment-16117273
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131762189
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java
 ---
@@ -291,6 +300,24 @@ public void run() {
 // Since all the runnables are now stopped, it is okay to stop the 
poller.
 stopPoller.shutdownNow();
 cleanupDir();
+if (eventHandler != null) {
+  if (stopStatus == null) {
+// if finalStatus is not set, the application must be stopped by a 
SystemMessages#STOP_COMMAND
+eventHandler.killed(twillRuntimeSpec.getTwillAppName(), 
twillRuntimeSpec.getTwillAppRunId());
+return;
+  }
+  switch (stopStatus) {
+case COMPLETED:
+  eventHandler.completed();
+  break;
+case ABORTED:
+eventHandler.aborted();
--- End diff --

Misalignment.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-240) Improve EventHandler to handle more application lifecycle events

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117275#comment-16117275
 ] 

ASF GitHub Bot commented on TWILL-240:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/58#discussion_r131762593
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/ProvisionTimeoutTestRun.java ---
@@ -85,13 +105,29 @@ public TimeoutAction 
launchTimeout(Iterable timeoutEvents) {
 return TimeoutAction.recheck(10, TimeUnit.SECONDS);
   }
 }
+
+@Override
+public void aborted() {
+  try {
+new 
File(context.getSpecification().getConfigs().get("parentFolderPath") + 
File.separator
--- End diff --

When constructing file, usually not to use `File.separator`, but instead 
use the constructor `File(File parent, String path)`.


> Improve EventHandler to handle more application lifecycle events
> 
>
> Key: TWILL-240
> URL: https://issues.apache.org/jira/browse/TWILL-240
> Project: Apache Twill
>  Issue Type: New Feature
>Reporter: Chengfeng Mao
>Assignee: Chengfeng Mao
> Fix For: 0.12.0
>
>
> Application Master should be able to run application specific code when 
> certain lifecycle events happen by calling methods from EventHandler. For 
> instance, when the app first starts, completes, aborts and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117509#comment-16117509
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/59
  
@anew Thanks for the review.


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-241) Allow specifying reserved off-heap memory and extra JVM options per runnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116858#comment-16116858
 ] 

ASF GitHub Bot commented on TWILL-241:
--

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/59#discussion_r131708600
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java
 ---
@@ -171,9 +168,27 @@ public String getKafkaZKConnect() {
   /**
* Returns the minimum heap ratio ({@link 
Configs.Keys#HEAP_RESERVED_MIN_RATIO}) based on the given configuration.
*/
-  private double getMinHeapRatio(Map config) {
-return config.containsKey(Configs.Keys.HEAP_RESERVED_MIN_RATIO) ?
-  Double.parseDouble(config.get(Configs.Keys.HEAP_RESERVED_MIN_RATIO)) 
:
-  Configs.Defaults.HEAP_RESERVED_MIN_RATIO;
+  private double getMinHeapRatio(@Nullable Map config, 
double defaultValue) {
+if (config == null) {
+  return defaultValue;
+}
+
+double ratio = 
config.containsKey(Configs.Keys.HEAP_RESERVED_MIN_RATIO) ?
+  Double.parseDouble(config.get(Configs.Keys.HEAP_RESERVED_MIN_RATIO)) 
: defaultValue;
+// Ratio can't be <= 0
+return ratio <= 0d ? defaultValue : ratio;
+  }
+
+  /**
+   * Returns the reserved memory size ({@link 
Configs.Keys#HEAP_RESERVED_MIN_RATIO}) based on the given configuration.
+   */
+  private int getReservedMemory(@Nullable Map config, int 
defaultValue) {
+if (config == null) {
+  return defaultValue;
--- End diff --

similar to getMinHeapRatio


> Allow specifying reserved off-heap memory and extra JVM options per runnable
> 
>
> Key: TWILL-241
> URL: https://issues.apache.org/jira/browse/TWILL-241
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, yarn
>Affects Versions: 0.11.0
>Reporter: Andreas Neumann
>Assignee: Terence Yim
> Fix For: 0.12.0
>
>
> Sometimes, a particular runnable needs a lot more off-heap memory than 
> others. It would therefore be useful to specify the amount of reserved 
> non-heap memory per runnable. 
> Similarly, for example, for debugging purposes, it may be necessary to add a 
> JVM option for one of the runnables without affecting the other runnables of 
> the same application.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-237) Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063371#comment-16063371
 ] 

ASF GitHub Bot commented on TWILL-237:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/55#discussion_r124058716
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/filesystem/FileContextLocation.java 
---
@@ -162,7 +163,8 @@ public URI toURI() {
 // append "port" to the path URI, while the DistributedFileSystem 
always use the cluster logical
 // name, which doesn't allow having port in it.
 URI uri = path.toUri();
-if (HAUtil.isLogicalUri(locationFactory.getConfiguration(), uri)) {
+
+if 
(FileContextLocationUtil.useLogicalUri(locationFactory.getConfiguration(), 
uri)) {
--- End diff --

It seems like we actually don't need to check, but rather just always strip 
off the port from the `URI`, since the `FileContext` or `FileSystem` in Hadoop 
will always determine it internally based on the `Configuration`.

In fact that would make the returning `URI` more predictable.


> Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8
> --
>
> Key: TWILL-237
> URL: https://issues.apache.org/jira/browse/TWILL-237
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>
> Twill is using hdfs.HAUtil apis that are suppose to be hdfs private and 
> subsequently signature of isLogicalURI was changed (actually name was 
> changed) in hadoop version 2.8
> Will post a patch for now to support both old and new names, but I think 
> eventually references to private hdfs interfaces/classes should be removed 
> from twill



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-237) Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063429#comment-16063429
 ] 

ASF GitHub Bot commented on TWILL-237:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/55#discussion_r124066638
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/filesystem/FileContextLocation.java 
---
@@ -162,7 +163,8 @@ public URI toURI() {
 // append "port" to the path URI, while the DistributedFileSystem 
always use the cluster logical
 // name, which doesn't allow having port in it.
 URI uri = path.toUri();
-if (HAUtil.isLogicalUri(locationFactory.getConfiguration(), uri)) {
+
+if 
(FileContextLocationUtil.useLogicalUri(locationFactory.getConfiguration(), 
uri)) {
--- End diff --

Yes. I think it should be safe to remove and always return an URI without 
port


> Twill is using hdfs HAUtil api that is nont-compatible with hadoop 2.8
> --
>
> Key: TWILL-237
> URL: https://issues.apache.org/jira/browse/TWILL-237
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>
> Twill is using hdfs.HAUtil apis that are suppose to be hdfs private and 
> subsequently signature of isLogicalURI was changed (actually name was 
> changed) in hadoop version 2.8
> Will post a patch for now to support both old and new names, but I think 
> eventually references to private hdfs interfaces/classes should be removed 
> from twill



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199301#comment-16199301
 ] 

ASF GitHub Bot commented on TWILL-248:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/62

(TWILL-248) Upgrade to use Netty-4.1

- Also enable ResourceReportClient to use HTTP compression

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-248-upgrade-netty-4.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/62.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #62


commit 321e7ba0116ccb7ba1c9b814ff60ad0dfd4ac3e5
Author: Terence Yim 
Date:   2017-10-10T20:26:11Z

(TWILL-248) Upgrade to use Netty-4.1

- Also enable ResourceReportClient to use HTTP compression




> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199326#comment-16199326
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user hsaputra commented on a diff in the pull request:

https://github.com/apache/twill/pull/62#discussion_r143848112
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/ResourceReportClient.java ---
@@ -52,12 +54,16 @@
   public ResourceReport get() {
 for (URL url : resourceUrls) {
   try {
-Reader reader = new BufferedReader(new 
InputStreamReader(url.openStream(), Charsets.UTF_8));
-try {
+HttpURLConnection urlConn = (HttpURLConnection) 
url.openConnection();
--- End diff --

Is this change related to Netty upgrade?


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199328#comment-16199328
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/62
  
One question about change form using BufferedReader,  but looks good.

If passes tests +1


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199368#comment-16199368
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/62#discussion_r143854276
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/ResourceReportClient.java ---
@@ -52,12 +54,16 @@
   public ResourceReport get() {
 for (URL url : resourceUrls) {
   try {
-Reader reader = new BufferedReader(new 
InputStreamReader(url.openStream(), Charsets.UTF_8));
-try {
+HttpURLConnection urlConn = (HttpURLConnection) 
url.openConnection();
--- End diff --

Not entirely. This is to enable compression when fetching the report.


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-244) Provide access to ResourceReport to TwillRunnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144636#comment-16144636
 ] 

ASF GitHub Bot commented on TWILL-244:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/61

(TWILL-244) Make resource report available to TwillRunnable

- Make resource report available to TwillContext
  - Enhance tracker service to provide more resource related endpoints
- Provide endpoints to fetch specific part of the report to reduce 
bandwidth
- Added a ResourceReporter API to provide programmatic access
  - Make resource report available via TwillContext
- Fixed a concurrent modification bug in ZKDiscoveryService.close

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-244-resource-report-runnable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/61.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #61


commit 53312962b33cc20c2f272b217d076730c498644d
Author: Terence Yim 
Date:   2017-08-28T23:47:45Z

(TWILL-244) Make resource report available to TwillContext

- Enhance tracker service to provide more resource related endpoints
  - Provide endpoints to fetch specific part of the report to reduce
bandwidth
  - Added a ResourceReporter API to provide programmatic access
- Make resource report available via TwillContext

commit 31facdf2f7f93f06c8fd131960be6d4c52fd646e
Author: Terence Yim 
Date:   2017-08-29T01:33:53Z

Fix a concurrent modification bug in ZKDiscoveryService.close




> Provide access to ResourceReport to TwillRunnable
> -
>
> Key: TWILL-244
> URL: https://issues.apache.org/jira/browse/TWILL-244
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Terence Yim
>
> One of the use case for this is to let TwillRunnable know about what 
> containers are running at the moment, which could be useful when building 
> fencing mechanism for single-master + HA type of applications.
> This information can be available via the TwillContext object.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-244) Provide access to ResourceReport to TwillRunnable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149820#comment-16149820
 ] 

ASF GitHub Bot commented on TWILL-244:
--

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/61#discussion_r136476455
  
--- Diff: 
twill-core/src/main/java/org/apache/twill/internal/ResourceReportClient.java ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.internal;
+
+import com.google.common.base.Charsets;
+import org.apache.twill.api.ResourceReport;
+import org.apache.twill.api.ResourceReporter;
+import org.apache.twill.api.TwillRunResources;
+import org.apache.twill.internal.json.ResourceReportAdapter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.lang.reflect.Type;
+import java.net.URL;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+/**
+ * A {@link ResourceReporter} that fetches reports from the given set of 
URLs.
+ */
+public final class ResourceReportClient implements ResourceReporter {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(ResourceReportClient.class);
+
+  private final List resourceUrls;
+
+  public ResourceReportClient(List resourceUrls) {
+this.resourceUrls = resourceUrls;
+  }
+
+  @Nullable
+  @Override
+  public ResourceReport getResourceReport() {
+for (URL url : resourceUrls) {
+  try {
+return fetchURL(url, "", ResourceReport.class);
+  } catch (IOException e) {
+// Just log a trace as it's ok to not able to fetch resource report
+LOG.trace("Exception raised when getting resource report from 
{}.", url, e);
+  }
+}
--- End diff --

even if all urls fail to fetch - why is that ok?


> Provide access to ResourceReport to TwillRunnable
> -
>
> Key: TWILL-244
> URL: https://issues.apache.org/jira/browse/TWILL-244
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api
>Reporter: Terence Yim
>
> One of the use case for this is to let TwillRunnable know about what 
> containers are running at the moment, which could be useful when building 
> fencing mechanism for single-master + HA type of applications.
> This information can be available via the TwillContext object.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-251) Hadoop21YarnNMClient logs too frequent on stopping container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277190#comment-16277190
 ] 

ASF GitHub Bot commented on TWILL-251:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/64


> Hadoop21YarnNMClient logs too frequent on stopping container
> 
>
> Key: TWILL-251
> URL: https://issues.apache.org/jira/browse/TWILL-251
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-251) Hadoop21YarnNMClient logs too frequent on stopping container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274815#comment-16274815
 ] 

ASF GitHub Bot commented on TWILL-251:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/64

(TWILL-251) Reduce log level of YarnNMClient

- Also reduce the polling frequency

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-251-reduce-log

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/64.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #64


commit 86a769db37650ff199ea5183f27c32dc0aa4cee3
Author: Terence Yim 
Date:   2017-12-01T19:21:07Z

(TWILL-251) Reduce log level of YarnNMClient

- Also reduce the polling frequency




> Hadoop21YarnNMClient logs too frequent on stopping container
> 
>
> Key: TWILL-251
> URL: https://issues.apache.org/jira/browse/TWILL-251
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-251) Hadoop21YarnNMClient logs too frequent on stopping container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274843#comment-16274843
 ] 

ASF GitHub Bot commented on TWILL-251:
--

Github user gsps1 commented on the issue:

https://github.com/apache/twill/pull/64
  
LGTM


> Hadoop21YarnNMClient logs too frequent on stopping container
> 
>
> Key: TWILL-251
> URL: https://issues.apache.org/jira/browse/TWILL-251
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225843#comment-16225843
 ] 

ASF GitHub Bot commented on TWILL-248:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/63

(TWILL-248) Speedup shutdown of tracker service



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill 
feature/TWILL-248-speedup-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/63.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #63


commit ef87c6210f01b5d4c24193b4e808220973e89cdc
Author: Terence Yim 
Date:   2017-10-30T22:10:34Z

(TWILL-248) Speedup shutdown of tracker service




> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227203#comment-16227203
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/63


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-248) Upgrade to latest Netty 4.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225851#comment-16225851
 ] 

ASF GitHub Bot commented on TWILL-248:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/62


> Upgrade to latest Netty 4.1
> ---
>
> Key: TWILL-248
> URL: https://issues.apache.org/jira/browse/TWILL-248
> Project: Apache Twill
>  Issue Type: Task
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
>
> Netty 3.x has been EOL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351800#comment-16351800
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845936
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

this should be protected not private


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351797#comment-16351797
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845875
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

why are you eliminating the hadoop-2.5 profile?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351798#comment-16351798
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845855
  
--- Diff: pom.xml ---
@@ -171,7 +171,7 @@
 4.11
 3.2
 1.5
-[2.0.2-alpha,2.3.0]
+2.7.2
 target/hadoop20-classes
--- End diff --

is there any reason to keep this property at all?  seems better to just 
have it defined in each profile as necessary.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351799#comment-16351799
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165845929
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

why remove final?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352347#comment-16352347
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user serranom commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165965620
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

Ah, okay then.  sorry, I missed that.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351922#comment-16351922
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862673
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

I didn't see any code specific to 2.5 that 2.3 did not provide?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351924#comment-16351924
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862857
  
--- Diff: 
twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAMClient.java
 ---
@@ -71,6 +73,11 @@ public Hadoop20YarnAMClient(Configuration conf) {
 this.nmClient = new Hadoop20YarnNMClient(YarnRPC.create(conf), conf);
   }
 
+  @Override
+  private ContainerId containerIdLookup(String containerIdStr) {
+return (ConverterUtils.toContainerId(containerIdStr));
--- End diff --

Thank you


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351925#comment-16351925
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r165862926
  
--- Diff: 
twill-yarn/src/main/hadoop22/org/apache/twill/internal/yarn/Hadoop22YarnAMClient.java
 ---
@@ -26,7 +26,7 @@
 /**
  * Wrapper class for AMRMClient for Hadoop version 2.2 or greater.
  */
-public final class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
+public class Hadoop22YarnAMClient extends Hadoop21YarnAMClient {
 
--- End diff --

If marked `final`, I can not extend it for 
[`Hadoop26YarnAMClient`](https://github.com/apache/twill/pull/65/files#diff-8c254de8ae00c6007495979dcb66a986R30)?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351984#comment-16351984
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
Thank you for the super fast code review @serranom! I think all comments 
have been addressed and the code should be complete now. Please let me know if 
you see anything else necessary. It is great working with Twill! Hopefully I 
can provide some other ideas as I work with it.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353173#comment-16353173
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r166160088
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

Yes, that's correct.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353176#comment-16353176
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/65
  
Changes LGTM.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353178#comment-16353178
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/65


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
> Fix For: 0.13.0
>
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351654#comment-16351654
 ] 

ASF GitHub Bot commented on TWILL-254:
--

GitHub user cbaenziger opened a pull request:

https://github.com/apache/twill/pull/65

(TWILL-254) Update to use ContainerId.fromString

This moves away from the deprecated `ConverterUitls.toContainerId` and 
updates the build to use Hadoop 2.7.2 to avoid folks grabbing master and having 
a failure trying to run on clusters with code newer than November 18th, 2014 
(release of Hadoop 2.6.0).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cbaenziger/twill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/65.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #65


commit 391553308ccc0539b72fac6bf13cd819ce3bb5e1
Author: Clay Baenziger 
Date:   2018-02-03T03:35:37Z

(TWILL-254) Update to use ContainerId.fromString




> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353027#comment-16353027
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on a diff in the pull request:

https://github.com/apache/twill/pull/65#discussion_r166133525
  
--- Diff: pom.xml ---
@@ -680,9 +680,9 @@
 
 
 
-hadoop-2.5
+hadoop-2.6
 
-2.5.1
+2.6.5
--- End diff --

@chtyim You are asking me to leave the 2.5 profile alone and add the 2.6 
profile?


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-254) Update to use ContainerId.fromString

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353022#comment-16353022
 ] 

ASF GitHub Bot commented on TWILL-254:
--

Github user cbaenziger commented on the issue:

https://github.com/apache/twill/pull/65
  
Hi @chtyim, thank you for the review. Could you please expand on what you 
see concerning in the `pom.xml`? For reference, I need to provide a 2.6 target 
as it is the first version to provide the `ContainerId.fromString()` API.


> Update to use ContainerId.fromString
> 
>
> Key: TWILL-254
> URL: https://issues.apache.org/jira/browse/TWILL-254
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.12.0
>Reporter: Clay B.
>Priority: Trivial
>
> Hello, trying to use 0.12 or master, I get the following on a Hortonworks 
> 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by 
> specifically building for Hadoop 2.6+ via {{mvn package 
> -Dhadoop.version=2.6.0}} but I was being a bit slow to realize what was up,) 
> I have a patch to move us away from the deprecated (see YARN-1942) call in 
> the stack trace anyhow and provide Hadoop 2.6 as a supported profile.
> {code:java}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
> Caused by: java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
>     at com.google.common.base.Throwables.propagate(Throwables.java:160)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
>     at 
> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
>     ... 5 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
>     ... 7 more
> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: 
> container_e164_1516738752188_0024_01_01
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>     at 
> org.apache.twill.internal.yarn.AbstractYarnAMClient.(AbstractYarnAMClient.java:83)
>     at 
> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.(Hadoop21YarnAMClient.java:65)
>     at 
> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.(Hadoop22YarnAMClient.java:34)
>     ... 12 more
> Caused by: java.lang.NumberFormatException: For input string: "e164"
>     at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>     at java.lang.Long.parseLong(Long.java:589)
>     at java.lang.Long.parseLong(Long.java:631)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>     at 
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>     ... 15 more
> {code}
> For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed 
> container ID formats and a new API call came out 
> [{{ContainerId.fromString}}|https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ContainerId.html#fromString(java.lang.String)].
>  Meanwhile, the container ID format change was not reflected, of course, in 
> older versions of Hadoop as clarified in YARN-3575.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-61) Second launch attempt of AM always failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393689#comment-16393689
 ] 

ASF GitHub Bot commented on TWILL-61:
-

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/67

(TWILL-61) Fix to allow higher attempts to relaunch the app after the first 
attempt failed

- Delete the Kafka root zk node for the application if already exist
- Delete the AM instance zk node if already exist
- For runnables parent zk node, it is not an error if it already exist
- Enhance KafkaClient publisher / consumer to deal with Kafka cluster 
changes
  - When AM killed and restarted, the embedded Kafka will be running in 
different host and port

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-61

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/67.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #67


commit 9decca071a9e067b30be2150a6097463c939b6af
Author: Terence Yim 
Date:   2018-03-09T20:21:26Z

(TWILL-61) Fix to allow higher attempts to relaunch the app after the first 
attempt failed

- Delete the Kafka root zk node for the application if already exist
- Delete the AM instance zk node if already exist
- For runnables parent zk node, it is not an error if it already exist
- Enhance KafkaClient publisher / consumer to deal with Kafka cluster 
changes
  - When AM killed and restarted, the embedded Kafka will be running in 
different host and port




> Second launch attempt of AM always failed
> -
>
> Key: TWILL-61
> URL: https://issues.apache.org/jira/browse/TWILL-61
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
>Priority: Major
> Fix For: 0.5.0-incubating
>
>
> YARN would make multiple attempts to launch an application. Currently second 
> or above attempts would always fail due to creation of /runId/state node in 
> ZK fail (node exists) because runId is generated on client side and doesn't 
> change between attempts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-255) incorrect logging after memory/cpu was adjusted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381313#comment-16381313
 ] 

ASF GitHub Bot commented on TWILL-255:
--

Github user hsaputra commented on the issue:

https://github.com/apache/twill/pull/66
  
+1
LGTM


> incorrect logging after memory/cpu was adjusted
> ---
>
> Key: TWILL-255
> URL: https://issues.apache.org/jira/browse/TWILL-255
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>Priority: Minor
>
> While adjusting resources for Containers when logging what was adjusted it 
> shows values after adjustment, so it's not known what it was adjusted from.
> Affected are: adjustCapability()
> Hadoop20YarnAMClient
> Hadoop21YarnAMClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TWILL-255) incorrect logging after memory/cpu was adjusted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381170#comment-16381170
 ] 

ASF GitHub Bot commented on TWILL-255:
--

GitHub user yufeldman opened a pull request:

https://github.com/apache/twill/pull/66

(TWILL-255) Incorrect logging after memory was adjusted. Does not sho…

…w memory before adjustment

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yufeldman/twill branch-TWILL-255

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/66.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #66


commit 5617478fc2ebf315ada125e1d3242f53c8545987
Author: Yuliya Feldman 
Date:   2018-02-28T22:52:56Z

(TWILL-255) Incorrect logging after memory was adjusted. Does not show 
memory before adjustment




> incorrect logging after memory/cpu was adjusted
> ---
>
> Key: TWILL-255
> URL: https://issues.apache.org/jira/browse/TWILL-255
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>Priority: Minor
>
> While adjusting resources for Containers when logging what was adjusted it 
> shows values after adjustment, so it's not known what it was adjusted from.
> Affected are: adjustCapability()
> Hadoop20YarnAMClient
> Hadoop21YarnAMClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)