date:20170403

[jira] [Resolved] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-230.
---
Resolution: Fixed

> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954531#comment-15954531
 ] 

ASF GitHub Bot commented on TWILL-230:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/53


> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #53: (TWILL-230) Get resource report based on the caller ...

2017-04-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/53


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954520#comment-15954520
 ] 

ASF GitHub Bot commented on TWILL-230:
--

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/53
  
LGTM


> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954391#comment-15954391
 ] 

ASF GitHub Bot commented on TWILL-230:
--

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/53
  
LGTM


> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill issue #53: (TWILL-230) Get resource report based on the caller user

2017-04-03 Thread anwar6953

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/53
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (TWILL-220) ResourceReportClient should not error log

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim reassigned TWILL-220:
-

Assignee: Terence Yim

> ResourceReportClient should not error log
> -
>
> Key: TWILL-220
> URL: https://issues.apache.org/jira/browse/TWILL-220
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Albert Shau
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> If the ResourceReportClient is unable to fetch the resource report, it logs 
> an error with a big stack trace. I have seen cluster setups where the 
> hostname are not set up correctly, so the call always fails. In those cases, 
> you end up with logs like:
> {noformat}
> 2017-02-24 06:45:09,542 - ERROR 
> [reporter-scheduler:o.a.t.y.ResourceReportClient@59] - Exception getting 
> resource report from http://xx:43931/resources.
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method) 
> ~[na:1.7.0_75]
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) 
> ~[na:1.7.0_75]
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>  ~[na:1.7.0_75]
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) 
> ~[na:1.7.0_75]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
> ~[na:1.7.0_75]
> at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_75]
> at java.net.Socket.connect(Socket.java:528) ~[na:1.7.0_75]
> at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.(HttpClient.java:211) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
>  ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
>  ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
>  ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
>  ~[na:1.7.0_75]
> at java.net.URL.openStream(URL.java:1037) ~[na:1.7.0_75]
> at 
> org.apache.twill.yarn.ResourceReportClient.get(ResourceReportClient.java:52) 
> ~[org.apache.twill.twill-yarn-0.9.0.jar:na]
> at 
> org.apache.twill.yarn.YarnTwillController.getResourceReport(YarnTwillController.java:330)
>  [co.cask.cdap.cdap-app-fabric-4.0.1.jar:na]
> at 
> co.cask.cdap.app.guice.ImpersonatedTwillController$1.call(ImpersonatedTwillController.java:86)
>  [na:na]
> at 
> co.cask.cdap.app.guice.ImpersonatedTwillController$1.call(ImpersonatedTwillController.java:82)
>  [na:na]
> at 
> co.cask.cdap.common.security.ImpersonationUtils$1.run(ImpersonationUtils.java:46)
>  [na:na]
> at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.7.0_75]
> at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_75]
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
>  [hadoop-common-2.7.1.2.4.2.0-258.jar:na]
> at 
> co.cask.cdap.common.security.ImpersonationUtils.doAs(ImpersonationUtils.java:43)
>  [na:na]
> at 
> co.cask.cdap.common.security.DefaultImpersonator.doAs(DefaultImpersonator.java:60)
>  [na:na]
> at 
> co.cask.cdap.app.guice.ImpersonatedTwillController.getResourceReport(ImpersonatedTwillController.java:82)
>  [na:na]
> at 
> co.cask.cdap.internal.app.runtime.distributed.DistributedProgramRuntimeService$ClusterResourceReporter.reportResources(DistributedProgramRuntimeService.java:385)
>  [na:na]
> at 
> co.cask.cdap.internal.app.runtime.AbstractResourceReporter.runOneIteration(AbstractResourceReporter.java:72)
>  [na:na]
> at 
> com.google.common.util.concurrent.AbstractScheduledService$1$1.run(AbstractScheduledService.java:170)
>  [com.google.guava.guava-13.0.1.jar:na]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
> [na:1.7.0_75]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
> [na:1.7.0_75]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  [na:1.7.0_75]
> at 
>

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954382#comment-15954382
 ] 

ASF GitHub Bot commented on TWILL-230:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/53#discussion_r109553180
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java ---
@@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) {
 
   @Override
   public ResourceReport getResourceReport() {
-// in case the user calls this before starting, return null
+// Only has resource report if the app is running.
+if (state() != State.RUNNING) {
+  return null;
+}
+ResourceReportClient resourcesClient = getResourcesClient();
 return (resourcesClient == null) ? null : resourcesClient.get();
   }
+
+  /**
+   * Returns the {@link ResourceReportClient} for fetching resource report 
from the AM.
+   * It first consults the RM for the tracking URL and get the resource 
report from there.
+   */
+  @Nullable
+  private ResourceReportClient getResourcesClient() {
+YarnApplicationReport report = processController.getReport();
+List urls = new ArrayList<>(2);
--- End diff --

Because we store the tracking URL and the original tracking URL.


> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (TWILL-220) ResourceReportClient should not error log

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-220:
--
Fix Version/s: 0.11.0

> ResourceReportClient should not error log
> -
>
> Key: TWILL-220
> URL: https://issues.apache.org/jira/browse/TWILL-220
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Albert Shau
> Fix For: 0.11.0
>
>
> If the ResourceReportClient is unable to fetch the resource report, it logs 
> an error with a big stack trace. I have seen cluster setups where the 
> hostname are not set up correctly, so the call always fails. In those cases, 
> you end up with logs like:
> {noformat}
> 2017-02-24 06:45:09,542 - ERROR 
> [reporter-scheduler:o.a.t.y.ResourceReportClient@59] - Exception getting 
> resource report from http://xx:43931/resources.
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method) 
> ~[na:1.7.0_75]
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) 
> ~[na:1.7.0_75]
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>  ~[na:1.7.0_75]
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) 
> ~[na:1.7.0_75]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
> ~[na:1.7.0_75]
> at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_75]
> at java.net.Socket.connect(Socket.java:528) ~[na:1.7.0_75]
> at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.(HttpClient.java:211) 
> ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[na:1.7.0_75]
> at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
>  ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
>  ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
>  ~[na:1.7.0_75]
> at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
>  ~[na:1.7.0_75]
> at java.net.URL.openStream(URL.java:1037) ~[na:1.7.0_75]
> at 
> org.apache.twill.yarn.ResourceReportClient.get(ResourceReportClient.java:52) 
> ~[org.apache.twill.twill-yarn-0.9.0.jar:na]
> at 
> org.apache.twill.yarn.YarnTwillController.getResourceReport(YarnTwillController.java:330)
>  [co.cask.cdap.cdap-app-fabric-4.0.1.jar:na]
> at 
> co.cask.cdap.app.guice.ImpersonatedTwillController$1.call(ImpersonatedTwillController.java:86)
>  [na:na]
> at 
> co.cask.cdap.app.guice.ImpersonatedTwillController$1.call(ImpersonatedTwillController.java:82)
>  [na:na]
> at 
> co.cask.cdap.common.security.ImpersonationUtils$1.run(ImpersonationUtils.java:46)
>  [na:na]
> at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.7.0_75]
> at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_75]
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
>  [hadoop-common-2.7.1.2.4.2.0-258.jar:na]
> at 
> co.cask.cdap.common.security.ImpersonationUtils.doAs(ImpersonationUtils.java:43)
>  [na:na]
> at 
> co.cask.cdap.common.security.DefaultImpersonator.doAs(DefaultImpersonator.java:60)
>  [na:na]
> at 
> co.cask.cdap.app.guice.ImpersonatedTwillController.getResourceReport(ImpersonatedTwillController.java:82)
>  [na:na]
> at 
> co.cask.cdap.internal.app.runtime.distributed.DistributedProgramRuntimeService$ClusterResourceReporter.reportResources(DistributedProgramRuntimeService.java:385)
>  [na:na]
> at 
> co.cask.cdap.internal.app.runtime.AbstractResourceReporter.runOneIteration(AbstractResourceReporter.java:72)
>  [na:na]
> at 
> com.google.common.util.concurrent.AbstractScheduledService$1$1.run(AbstractScheduledService.java:170)
>  [com.google.guava.guava-13.0.1.jar:na]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
> [na:1.7.0_75]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
> [na:1.7.0_75]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  [na:1.7.0_75]
> at 
>

[GitHub] twill pull request #53: (TWILL-230) Get resource report based on the caller ...

2017-04-03 Thread chtyim

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/53#discussion_r109553180
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java ---
@@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) {
 
   @Override
   public ResourceReport getResourceReport() {
-// in case the user calls this before starting, return null
+// Only has resource report if the app is running.
+if (state() != State.RUNNING) {
+  return null;
+}
+ResourceReportClient resourcesClient = getResourcesClient();
 return (resourcesClient == null) ? null : resourcesClient.get();
   }
+
+  /**
+   * Returns the {@link ResourceReportClient} for fetching resource report 
from the AM.
+   * It first consults the RM for the tracking URL and get the resource 
report from there.
+   */
+  @Nullable
+  private ResourceReportClient getResourcesClient() {
+YarnApplicationReport report = processController.getReport();
+List urls = new ArrayList<>(2);
--- End diff --

Because we store the tracking URL and the original tracking URL.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954373#comment-15954373
 ] 

ASF GitHub Bot commented on TWILL-230:
--

Github user yaojiefeng commented on a diff in the pull request:

https://github.com/apache/twill/pull/53#discussion_r109552373
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java ---
@@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) {
 
   @Override
   public ResourceReport getResourceReport() {
-// in case the user calls this before starting, return null
+// Only has resource report if the app is running.
+if (state() != State.RUNNING) {
+  return null;
+}
+ResourceReportClient resourcesClient = getResourcesClient();
 return (resourcesClient == null) ? null : resourcesClient.get();
   }
+
+  /**
+   * Returns the {@link ResourceReportClient} for fetching resource report 
from the AM.
+   * It first consults the RM for the tracking URL and get the resource 
report from there.
+   */
+  @Nullable
+  private ResourceReportClient getResourcesClient() {
+YarnApplicationReport report = processController.getReport();
+List urls = new ArrayList<>(2);
--- End diff --

Why will this url list has initial capacity of 2?


> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #53: (TWILL-230) Get resource report based on the caller ...

2017-04-03 Thread yaojiefeng

Github user yaojiefeng commented on a diff in the pull request:

https://github.com/apache/twill/pull/53#discussion_r109552373
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java ---
@@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) {
 
   @Override
   public ResourceReport getResourceReport() {
-// in case the user calls this before starting, return null
+// Only has resource report if the app is running.
+if (state() != State.RUNNING) {
+  return null;
+}
+ResourceReportClient resourcesClient = getResourcesClient();
 return (resourcesClient == null) ? null : resourcesClient.get();
   }
+
+  /**
+   * Returns the {@link ResourceReportClient} for fetching resource report 
from the AM.
+   * It first consults the RM for the tracking URL and get the resource 
report from there.
+   */
+  @Nullable
+  private ResourceReportClient getResourcesClient() {
+YarnApplicationReport report = processController.getReport();
+List urls = new ArrayList<>(2);
--- End diff --

Why will this url list has initial capacity of 2?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954351#comment-15954351
 ] 

ASF GitHub Bot commented on TWILL-230:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/53

(TWILL-230) Get resource report based on the caller user

- Also by default get the resource report from the tracking url, then fall 
back to the original tracking url.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-230

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/53.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #53


commit 239619d8a1ccb08dae3c3262f380b87759a574f6
Author: Terence Yim 
Date:   2017-04-03T23:49:52Z

(TWILL-230) Get resource report based on the caller user

- Also by default get the resource report from the tracking url, then fall 
back to the original tracking url.




> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #52: (TWILL-176) Redirect all requests to tracker URL to ...

2017-04-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/52


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-176) Set ResourceManager tracking URL to resources path.

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954257#comment-15954257
 ] 

ASF GitHub Bot commented on TWILL-176:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/52


> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (TWILL-176) Set ResourceManager tracking URL to resources path.

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-176.
---
Resolution: Fixed

> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread Terence Yim (JIRA)

Terence Yim created TWILL-230:
-

 Summary: Retrieval of ResourceReport via TwillController may fail
 Key: TWILL-230
 URL: https://issues.apache.org/jira/browse/TWILL-230
 Project: Apache Twill
  Issue Type: Bug
Reporter: Terence Yim


When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
applications and get the tracker URL from the application report. However, the 
operation might fail if the application was started with a different user than 
the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-230:
--
Fix Version/s: 0.11.0

> Retrieval of ResourceReport via TwillController may fail
> 
>
> Key: TWILL-230
> URL: https://issues.apache.org/jira/browse/TWILL-230
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> When starting a new {{YarnTwillRunnerService}}, it discovers all existing 
> applications and get the tracker URL from the application report. However, 
> the operation might fail if the application was started with a different user 
> than the one who starts the {{YarnTwillRunnerService}} in a Kerberos cluster, 
> resulting in failure to get resource report.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (TWILL-176) Set ResourceManager tracking URL to resources path.

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim reassigned TWILL-176:
-

Assignee: Terence Yim

> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (TWILL-176) Set ResourceManager tracking URL to resources path.

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-176:
--
Fix Version/s: 0.11.0

> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
> Fix For: 0.11.0
>
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-176) Set ResourceManager tracking URL to resources path.

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954214#comment-15954214
 ] 

ASF GitHub Bot commented on TWILL-176:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/52

(TWILL-176) Redirect all requests to tracker URL to /resources path

- "/resources" is the only path supported for the tracker service for now

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-176

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/52.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #52


commit b07fd47e3e78ad80097e26a944baceebefd0559c
Author: Terence Yim 
Date:   2017-04-03T21:42:44Z

(TWILL-176) Redirect all requests to tracker URL to /resources path

- "/resources" is the only path supported for the tracker service for now




> Set ResourceManager tracking URL to resources path.
> ---
>
> Key: TWILL-176
> URL: https://issues.apache.org/jira/browse/TWILL-176
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Reporter: Albert Shau
>
> The resource manager tracking url is currently set to the host and port only. 
> There is nothing exposing the fact that you can append 'resources' to the 
> path to get a list of containers and information about each container.
> This was done because the YARN api was to working properly at the time, but 
> we should re-investigate to see if this is still the case, or perhaps serve 
> some base page there that links to the resources page, plus anything else we 
> decide to add in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #52: (TWILL-176) Redirect all requests to tracker URL to ...

2017-04-03 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/52

(TWILL-176) Redirect all requests to tracker URL to /resources path

- "/resources" is the only path supported for the tracker service for now

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-176

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/52.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #52


commit b07fd47e3e78ad80097e26a944baceebefd0559c
Author: Terence Yim 
Date:   2017-04-03T21:42:44Z

(TWILL-176) Redirect all requests to tracker URL to /resources path

- "/resources" is the only path supported for the tracker service for now




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (TWILL-189) SecureStoreUpdater should support updating secure store as other users

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-189.
---
Resolution: Fixed

> SecureStoreUpdater should support updating secure store as other users
> --
>
> Key: TWILL-189
> URL: https://issues.apache.org/jira/browse/TWILL-189
> Project: Apache Twill
>  Issue Type: New Feature
>  Components: api
>Affects Versions: 0.7.0-incubating
>Reporter: Ali Anwar
> Fix For: 0.11.0
>
>
> SecureStoreUpdater always creates secure stores for the current user and 
> writes these as the current user.
> It should allow writing the secure store as a different user.
> For an example use case, see: https://github.com/caskdata/cdap/pull/6380



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954094#comment-15954094
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/51


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (TWILL-228) Application jar is not properly built when the client bundles application and twill classes in the same fat jar

2017-04-03 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-228.
---
Resolution: Fixed

> Application jar is not properly built when the client bundles application and 
> twill classes in the same fat jar
> ---
>
> Key: TWILL-228
> URL: https://issues.apache.org/jira/browse/TWILL-228
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.10.0
>Reporter: Sam William
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The actual error the client sees is
> Application application_1484158548936_11154 failed 2 times due to AM 
> Container for appattempt_1484158548936_11154_02 exited with exitCode: 
> -1000
> For more detailed output, check application tracking page<> Then, click on 
> links to logs of each attempt.
> Diagnostics: No such file or directory
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
> at 
> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at 
> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Failing this attempt. Failing the application.
> Digging in a little bit, it appears this happens when all the application 
> classes and twill classes and bundled together in the same fat jar.   This 
> workaround helped 
> https://github.com/sampd/twill/compare/branch-0.10.0...sampd:v10_test?expand=1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #51: (TWILL-179) Added support for custom ClassLoader for...

2017-04-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/51


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954090#comment-15954090
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/51#discussion_r109512583
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/CustomClassLoader.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import org.apache.twill.api.ServiceAnnouncer;
+import org.apache.twill.common.Cancellable;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+import org.objectweb.asm.commons.GeneratorAdapter;
+import org.objectweb.asm.commons.Method;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * ClassLoader that generates a new class for the {@link 
CustomClassLoaderTestRun}.
+ */
+public final class CustomClassLoader extends URLClassLoader {
+
+  public CustomClassLoader(URL[] urls, ClassLoader parent) {
+super(urls, parent);
+  }
+
+  @Override
+  protected Class findClass(String name) throws ClassNotFoundException {
+if (!CustomClassLoaderRunnable.GENERATED_CLASS_NAME.equals(name)) {
+  return super.findClass(name);
+}
+
+// Generate a class that look like this:
+//
+// public class Generated {
+//
+//   public void announce(ServiceAnnouncer announcer, String 
serviceName, int port) {
--- End diff --

actually the comment is correct. The generated method has `void` return 
type. The `announcer.announce` method has `Cancellable` return type, but the 
result gets ignored (hence the `mg.pop()` code in line 79).


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954088#comment-15954088
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/51
  
1 minor comment; remainder LGTM.


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill issue #51: (TWILL-179) Added support for custom ClassLoader for contai...

2017-04-03 Thread anwar6953

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/51
  
1 minor comment; remainder LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954077#comment-15954077
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/51#discussion_r109510575
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/CustomClassLoader.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import org.apache.twill.api.ServiceAnnouncer;
+import org.apache.twill.common.Cancellable;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+import org.objectweb.asm.commons.GeneratorAdapter;
+import org.objectweb.asm.commons.Method;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * ClassLoader that generates a new class for the {@link 
CustomClassLoaderTestRun}.
+ */
+public final class CustomClassLoader extends URLClassLoader {
+
+  public CustomClassLoader(URL[] urls, ClassLoader parent) {
+super(urls, parent);
+  }
+
+  @Override
+  protected Class findClass(String name) throws ClassNotFoundException {
+if (!CustomClassLoaderRunnable.GENERATED_CLASS_NAME.equals(name)) {
+  return super.findClass(name);
+}
+
+// Generate a class that look like this:
+//
+// public class Generated {
+//
+//   public void announce(ServiceAnnouncer announcer, String 
serviceName, int port) {
--- End diff --

Will update.


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #51: (TWILL-179) Added support for custom ClassLoader for...

2017-04-03 Thread anwar6953

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/51#discussion_r109510121
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/CustomClassLoader.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import org.apache.twill.api.ServiceAnnouncer;
+import org.apache.twill.common.Cancellable;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+import org.objectweb.asm.commons.GeneratorAdapter;
+import org.objectweb.asm.commons.Method;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * ClassLoader that generates a new class for the {@link 
CustomClassLoaderTestRun}.
+ */
+public final class CustomClassLoader extends URLClassLoader {
+
+  public CustomClassLoader(URL[] urls, ClassLoader parent) {
+super(urls, parent);
+  }
+
+  @Override
+  protected Class findClass(String name) throws ClassNotFoundException {
+if (!CustomClassLoaderRunnable.GENERATED_CLASS_NAME.equals(name)) {
+  return super.findClass(name);
+}
+
+// Generate a class that look like this:
+//
+// public class Generated {
+//
+//   public void announce(ServiceAnnouncer announcer, String 
serviceName, int port) {
--- End diff --

Is the javadoc outdated/inconsistent?
`Cancellable` is declared as the return type, below on line 76.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954075#comment-15954075
 ] 

ASF GitHub Bot commented on TWILL-179:
--

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/51#discussion_r109510121
  
--- Diff: 
twill-yarn/src/test/java/org/apache/twill/yarn/CustomClassLoader.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.yarn;
+
+import org.apache.twill.api.ServiceAnnouncer;
+import org.apache.twill.common.Cancellable;
+import org.objectweb.asm.ClassWriter;
+import org.objectweb.asm.Opcodes;
+import org.objectweb.asm.Type;
+import org.objectweb.asm.commons.GeneratorAdapter;
+import org.objectweb.asm.commons.Method;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * ClassLoader that generates a new class for the {@link 
CustomClassLoaderTestRun}.
+ */
+public final class CustomClassLoader extends URLClassLoader {
+
+  public CustomClassLoader(URL[] urls, ClassLoader parent) {
+super(urls, parent);
+  }
+
+  @Override
+  protected Class findClass(String name) throws ClassNotFoundException {
+if (!CustomClassLoaderRunnable.GENERATED_CLASS_NAME.equals(name)) {
+  return super.findClass(name);
+}
+
+// Generate a class that look like this:
+//
+// public class Generated {
+//
+//   public void announce(ServiceAnnouncer announcer, String 
serviceName, int port) {
--- End diff --

Is the javadoc outdated/inconsistent?
`Cancellable` is declared as the return type, below on line 76.


> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #51: (TWILL-179) Added support for custom ClassLoader for...

2017-04-03 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/51

(TWILL-179) Added support for custom ClassLoader for containers

- Added method TwillPreparer.setClassLoader
- Use system property "twill.custom.class.loader" to pass the class name of 
the custom ClassLoader

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-179

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/51.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #51


commit 955156ee8b1cfb8e4072f77c9f83ee4ebce2e03b
Author: Terence Yim 
Date:   2017-04-03T19:34:14Z

(TWILL-179) Added support for custom ClassLoader for containers

- Added method TwillPreparer.setClassLoader
- Use system property "twill.custom.class.loader" to pass the class name of 
the custom ClassLoader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954050#comment-15954050
 ] 

ASF GitHub Bot commented on TWILL-179:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/51

(TWILL-179) Added support for custom ClassLoader for containers

- Added method TwillPreparer.setClassLoader
- Use system property "twill.custom.class.loader" to pass the class name of 
the custom ClassLoader

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-179

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/51.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #51


commit 955156ee8b1cfb8e4072f77c9f83ee4ebce2e03b
Author: Terence Yim 
Date:   2017-04-03T19:34:14Z

(TWILL-179) Added support for custom ClassLoader for containers

- Added method TwillPreparer.setClassLoader
- Use system property "twill.custom.class.loader" to pass the class name of 
the custom ClassLoader




> Provides more flexibility in controlling the container classpath and 
> classloader
> 
>
> Key: TWILL-179
> URL: https://issues.apache.org/jira/browse/TWILL-179
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently the user has little control on the classpath and classloader of the 
> runnable container. E.g. in Java, one can control both through the {{-cp}} 
> and {{-Djava.system.class.loader}} arguments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[GitHub] twill pull request #53: (TWILL-230) Get resource report based on the caller ...

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[GitHub] twill issue #53: (TWILL-230) Get resource report based on the caller user

[jira] [Assigned] (TWILL-220) ResourceReportClient should not error log

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[jira] [Updated] (TWILL-220) ResourceReportClient should not error log

[GitHub] twill pull request #53: (TWILL-230) Get resource report based on the caller ...

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[GitHub] twill pull request #53: (TWILL-230) Get resource report based on the caller ...

[jira] [Commented] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[GitHub] twill pull request #52: (TWILL-176) Redirect all requests to tracker URL to ...

[jira] [Commented] (TWILL-176) Set ResourceManager tracking URL to resources path.

[jira] [Resolved] (TWILL-176) Set ResourceManager tracking URL to resources path.

[jira] [Created] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[jira] [Updated] (TWILL-230) Retrieval of ResourceReport via TwillController may fail

[jira] [Assigned] (TWILL-176) Set ResourceManager tracking URL to resources path.

[jira] [Updated] (TWILL-176) Set ResourceManager tracking URL to resources path.

[jira] [Commented] (TWILL-176) Set ResourceManager tracking URL to resources path.

[GitHub] twill pull request #52: (TWILL-176) Redirect all requests to tracker URL to ...

[jira] [Resolved] (TWILL-189) SecureStoreUpdater should support updating secure store as other users

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[jira] [Resolved] (TWILL-228) Application jar is not properly built when the client bundles application and twill classes in the same fat jar

[GitHub] twill pull request #51: (TWILL-179) Added support for custom ClassLoader for...

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[GitHub] twill issue #51: (TWILL-179) Added support for custom ClassLoader for contai...

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[GitHub] twill pull request #51: (TWILL-179) Added support for custom ClassLoader for...

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

[GitHub] twill pull request #51: (TWILL-179) Added support for custom ClassLoader for...

[jira] [Commented] (TWILL-179) Provides more flexibility in controlling the container classpath and classloader

34 matches

Site Navigation

Mail list logo

Footer information