[jira] [Commented] (TWILL-122) Allow disabling the log transport

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935331#comment-15935331
 ] 

ASF GitHub Bot commented on TWILL-122:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/40

(TWILL-122) Allow disabling log collection

- Introduced a new configuration twill.log.collection.enabled for
  turning off log collection
- Refactor YarnTwillController and related class hierarchy to not
  starting Kafka client when log collection is disabled
- Added Kafka zk connection string information in AM live node data
- Refactor KafkaAppender and ServiceMain configureLogger
  - Log to StatusManager instead of Logger to avoid recursive logging
  - Instead of resetting logback configuration, directly instantiate and
add the Kafka log appender to the logging context.
- Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to
  simplify ZK Connection string construction

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-122

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/40.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #40


commit 617280ee72b2e7fc7a3cc473d6dfe35576cf5e57
Author: Terence Yim 
Date:   2017-03-20T21:42:46Z

(TWILL-122) Allow disabling log collection

- Introduced a new configuration twill.log.collection.enabled for
  turning off log collection
- Refactor YarnTwillController and related class hierarchy to not
  starting Kafka client when log collection is disabled
- Added Kafka zk connection string information in AM live node data
- Refactor KafkaAppender and ServiceMain configureLogger
  - Log to StatusManager instead of Logger to avoid recursive logging
  - Instead of resetting logback configuration, directly instantiate and
add the Kafka log appender to the logging context.
- Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to
  simplify ZK Connection string construction




> Allow disabling the log transport
> -
>
> Key: TWILL-122
> URL: https://issues.apache.org/jira/browse/TWILL-122
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: core
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently transporting logs to Kafka is mandatory. It should be optionally so 
> that application can have its own way of log collection.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #40: (TWILL-122) Allow disabling log collection

[chtyim]

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/40

(TWILL-122) Allow disabling log collection

- Introduced a new configuration twill.log.collection.enabled for
  turning off log collection
- Refactor YarnTwillController and related class hierarchy to not
  starting Kafka client when log collection is disabled
- Added Kafka zk connection string information in AM live node data
- Refactor KafkaAppender and ServiceMain configureLogger
  - Log to StatusManager instead of Logger to avoid recursive logging
  - Instead of resetting logback configuration, directly instantiate and
add the Kafka log appender to the logging context.
- Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to
  simplify ZK Connection string construction

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-122

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/40.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #40


commit 617280ee72b2e7fc7a3cc473d6dfe35576cf5e57
Author: Terence Yim 
Date:   2017-03-20T21:42:46Z

(TWILL-122) Allow disabling log collection

- Introduced a new configuration twill.log.collection.enabled for
  turning off log collection
- Refactor YarnTwillController and related class hierarchy to not
  starting Kafka client when log collection is disabled
- Added Kafka zk connection string information in AM live node data
- Refactor KafkaAppender and ServiceMain configureLogger
  - Log to StatusManager instead of Logger to avoid recursive logging
  - Instead of resetting logback configuration, directly instantiate and
add the Kafka log appender to the logging context.
- Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to
  simplify ZK Connection string construction




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/39


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-225) Allow using different configurations per application submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935255#comment-15935255
 ] 

ASF GitHub Bot commented on TWILL-225:
--

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/39#discussion_r107265236
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, 
Map logL
 this.logLevels.put(runnableName, newLevels);
   }
 
+  /**
+   * Creates an {@link Credentials} by copying the {@link Credentials} of 
the current user.
+   */
   private Credentials createCredentials() {
 Credentials credentials = new Credentials();
 
 try {
   
credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials());
+} catch (IOException e) {
--- End diff --

I guess app submission would fail if caller doesn't acquire required 
credentials in its own way anyways.


> Allow using different configurations per application submission
> ---
>
> Key: TWILL-225
> URL: https://issues.apache.org/jira/browse/TWILL-225
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently there are couple configurations that can be provided via the hadoop 
> {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those 
> configurations are global (same for all app launched through the same 
> {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes 
> method to alter the configuration for a given app submission.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-225) Allow using different configurations per application submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935253#comment-15935253
 ] 

ASF GitHub Bot commented on TWILL-225:
--

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/39
  
LGTM


> Allow using different configurations per application submission
> ---
>
> Key: TWILL-225
> URL: https://issues.apache.org/jira/browse/TWILL-225
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently there are couple configurations that can be provided via the hadoop 
> {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those 
> configurations are global (same for all app launched through the same 
> {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes 
> method to alter the configuration for a given app submission.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...

[anwar6953]

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/39#discussion_r107265236
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, 
Map logL
 this.logLevels.put(runnableName, newLevels);
   }
 
+  /**
+   * Creates an {@link Credentials} by copying the {@link Credentials} of 
the current user.
+   */
   private Credentials createCredentials() {
 Credentials credentials = new Credentials();
 
 try {
   
credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials());
+} catch (IOException e) {
--- End diff --

I guess app submission would fail if caller doesn't acquire required 
credentials in its own way anyways.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill issue #39: (TWILL-225) Refactor TwillPreprer to allow configurations a...

[anwar6953]

Github user anwar6953 commented on the issue:

https://github.com/apache/twill/pull/39
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-225) Allow using different configurations per application submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935247#comment-15935247
 ] 

ASF GitHub Bot commented on TWILL-225:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/39#discussion_r107264780
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, 
Map logL
 this.logLevels.put(runnableName, newLevels);
   }
 
+  /**
+   * Creates an {@link Credentials} by copying the {@link Credentials} of 
the current user.
+   */
   private Credentials createCredentials() {
 Credentials credentials = new Credentials();
 
 try {
   
credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials());
+} catch (IOException e) {
--- End diff --

I don't think so. Even if this call fail, if the caller is able to acquire 
required credentials in his own way and add them via the `addSecureStore` 
method, then the app can still be submitted and executed correctly.


> Allow using different configurations per application submission
> ---
>
> Key: TWILL-225
> URL: https://issues.apache.org/jira/browse/TWILL-225
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently there are couple configurations that can be provided via the hadoop 
> {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those 
> configurations are global (same for all app launched through the same 
> {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes 
> method to alter the configuration for a given app submission.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...

[chtyim]

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/39#discussion_r107264780
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, 
Map logL
 this.logLevels.put(runnableName, newLevels);
   }
 
+  /**
+   * Creates an {@link Credentials} by copying the {@link Credentials} of 
the current user.
+   */
   private Credentials createCredentials() {
 Credentials credentials = new Credentials();
 
 try {
   
credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials());
+} catch (IOException e) {
--- End diff --

I don't think so. Even if this call fail, if the caller is able to acquire 
required credentials in his own way and add them via the `addSecureStore` 
method, then the app can still be submitted and executed correctly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-225) Allow using different configurations per application submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935227#comment-15935227
 ] 

ASF GitHub Bot commented on TWILL-225:
--

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/39#discussion_r107262119
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, 
Map logL
 this.logLevels.put(runnableName, newLevels);
   }
 
+  /**
+   * Creates an {@link Credentials} by copying the {@link Credentials} of 
the current user.
+   */
   private Credentials createCredentials() {
 Credentials credentials = new Credentials();
 
 try {
   
credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials());
+} catch (IOException e) {
--- End diff --

shouldn't this be fatal?


> Allow using different configurations per application submission
> ---
>
> Key: TWILL-225
> URL: https://issues.apache.org/jira/browse/TWILL-225
> Project: Apache Twill
>  Issue Type: Improvement
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> Currently there are couple configurations that can be provided via the hadoop 
> {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those 
> configurations are global (same for all app launched through the same 
> {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes 
> method to alter the configuration for a given app submission.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...

[anwar6953]

Github user anwar6953 commented on a diff in the pull request:

https://github.com/apache/twill/pull/39#discussion_r107262119
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java ---
@@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, 
Map logL
 this.logLevels.put(runnableName, newLevels);
   }
 
+  /**
+   * Creates an {@link Credentials} by copying the {@link Credentials} of 
the current user.
+   */
   private Credentials createCredentials() {
 Credentials credentials = new Credentials();
 
 try {
   
credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials());
+} catch (IOException e) {
--- End diff --

shouldn't this be fatal?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---