date:20170327

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Terence Yim

Hi Sam,

Do you have twill classes included in the fat jar as well? I think it might be 
affecting the application jar that twill generate. Would you mind filing a JIRA 
on this?

Thanks,
Terence

> On Mar 27, 2017, at 3:55 PM, Sam William  wrote:
> 
> I was able to make it work with this 1 line change. 
> 
> https://github.com/sampd/twill/compare/branch-0.10.0...sampd:v10_test?expand=1
>  
> 
> 
> We have a fat jar with all dependencies included for the application. Could 
> that be the cause ?
> 
> Sam
> 
> 
> 
>> On Mar 27, 2017, at 15:49, Yuliya Feldman  wrote:
>> 
>> Code of your application you want to be running in YARN I believe :)
>> 
>> On Mon, Mar 27, 2017 at 3:28 PM, Sam William  wrote:
>> 
>>> Yes. 22 bytes looks like an empty zip file.  Any idea what should there in
>>> the application jar file ?
>>> 
>>> Sam
 On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
 
 File is very small - it may be nothing to do with file not found. Either
 permissions or something else
 
 On Mon, Mar 27, 2017 at 1:17 PM, Sam William 
>>> wrote:
 
> I logged into the master host and looked at the nodemanager logs. It
>>> fails
> at localizing the application jar.  The files are there in HDFS.  I can
> even see it is able to copy the other files just fine (for example the
> launcher jar and runtime.config)
> 
> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>>> 44a506886fc1/launcher.
> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
> 
> 
> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.localizer.LocalizedResource: Resource
> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
> appcache/application_1484158548936_11282/filecache/
> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
>>> transitioned
> from DOWNLOADING to LOCALIZED
> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.localizer.LocalizedResource: Resource
> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
> appcache/application_1484158548936_11282/filecache/
> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
> DOWNLOADING to LOCALIZED
> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.
>>> UserGroupInformation:
> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such
>>> file
> or directory
> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
> nodemanager.containermanager.localizer.ResourceLocalizationService: {
> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-
>>> 4121-824e-448a12a635c1.jar,
> 1490644063924, ARCHIVE, null } failed: No such file or directory
> ENOENT: No such file or directory
>  at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
> Method)
>  at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
> java:230)
>  at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> RawLocalFileSystem.java:660)
>  at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
> DelegateToFileSystem.java:206)
>  at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:
>>> 251)
>  at org.apache.hadoop.fs.FileContext$10.next(
>>> FileContext.java:955)
>  at org.apache.hadoop.fs.FileContext$10.next(
>>> FileContext.java:951)
>  at or

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944425#comment-15944425
 ] 

ASF GitHub Bot commented on TWILL-227:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/46
  
This replaces #45 due to accidentally deleting the branch.


> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill issue #46: (TWILL-227) Disabling caching of FileSystem instance when g...

2017-03-27 Thread chtyim

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/46
  
This replaces #45 due to accidentally deleting the branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill pull request #46: (TWILL-227) Disabling caching of FileSystem instance...

2017-03-27 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/46

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-227

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/46.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #46


commit b2ce61f9023770aab9efd180b8e662212784d137
Author: Terence Yim 
Date:   2017-03-27T23:37:07Z

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944424#comment-15944424
 ] 

ASF GitHub Bot commented on TWILL-227:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/46

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-227

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/46.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #46


commit b2ce61f9023770aab9efd180b8e662212784d137
Author: Terence Yim 
Date:   2017-03-27T23:37:07Z

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token




> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944422#comment-15944422
 ] 

ASF GitHub Bot commented on TWILL-227:
--

Github user chtyim closed the pull request at:

https://github.com/apache/twill/pull/45


> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #45: (TWILL-227) Disabling caching of FileSystem instance...

2017-03-27 Thread chtyim

Github user chtyim closed the pull request at:

https://github.com/apache/twill/pull/45


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944262#comment-15944262
 ] 

ASF GitHub Bot commented on TWILL-227:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/45

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-227

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/45.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #45


commit b2ce61f9023770aab9efd180b8e662212784d137
Author: Terence Yim 
Date:   2017-03-27T23:37:07Z

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token




> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #45: (TWILL-227) Disabling caching of FileSystem instance...

2017-03-27 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/45

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-227

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/45.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #45


commit b2ce61f9023770aab9efd180b8e662212784d137
Author: Terence Yim 
Date:   2017-03-27T23:37:07Z

(TWILL-227) Disabling caching of FileSystem instance when getting 
delegation token




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (TWILL-226) Remove the deprecated HDFSLocationFactory and related classes

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-226.
---
Resolution: Fixed

> Remove the deprecated HDFSLocationFactory and related classes
> -
>
> Key: TWILL-226
> URL: https://issues.apache.org/jira/browse/TWILL-226
> Project: Apache Twill
>  Issue Type: Task
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-226) Remove the deprecated HDFSLocationFactory and related classes

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944254#comment-15944254
 ] 

ASF GitHub Bot commented on TWILL-226:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/44


> Remove the deprecated HDFSLocationFactory and related classes
> -
>
> Key: TWILL-226
> URL: https://issues.apache.org/jira/browse/TWILL-226
> Project: Apache Twill
>  Issue Type: Task
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #44: (TWILL-226) Remove the deprecated HDFSLocationFactor...

2017-03-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/44


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-226) Remove the deprecated HDFSLocationFactory and related classes

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944218#comment-15944218
 ] 

ASF GitHub Bot commented on TWILL-226:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/44

(TWILL-226) Remove the deprecated HDFSLocationFactory



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-226

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/44.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #44


commit dcb9fb9fcb853dbee65682fbc15b214521a89434
Author: Terence Yim 
Date:   2017-03-27T23:14:17Z

(TWILL-226) Remove the deprecated HDFSLocationFactory




> Remove the deprecated HDFSLocationFactory and related classes
> -
>
> Key: TWILL-226
> URL: https://issues.apache.org/jira/browse/TWILL-226
> Project: Apache Twill
>  Issue Type: Task
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-227:
--
Fix Version/s: 0.11.0

> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #44: (TWILL-226) Remove the deprecated HDFSLocationFactor...

2017-03-27 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/44

(TWILL-226) Remove the deprecated HDFSLocationFactory



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-226

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/44.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #44


commit dcb9fb9fcb853dbee65682fbc15b214521a89434
Author: Terence Yim 
Date:   2017-03-27T23:14:17Z

(TWILL-226) Remove the deprecated HDFSLocationFactory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

2017-03-27 Thread Terence Yim (JIRA)

Terence Yim created TWILL-227:
-

 Summary: Possible leakage of FileSystem object when 
YarnUtils.addDelegationTokens is called with different user
 Key: TWILL-227
 URL: https://issues.apache.org/jira/browse/TWILL-227
 Project: Apache Twill
  Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Terence Yim
Assignee: Terence Yim


The {{FileSystem.get}} caches instance for different UGI by default and the 
cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-226) Remove the deprecated HDFSLocationFactory and related classes

2017-03-27 Thread Terence Yim (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944199#comment-15944199
 ] 

Terence Yim commented on TWILL-226:
---

The class has been deprecated for a long time. It should get removed.

> Remove the deprecated HDFSLocationFactory and related classes
> -
>
> Key: TWILL-226
> URL: https://issues.apache.org/jira/browse/TWILL-226
> Project: Apache Twill
>  Issue Type: Task
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (TWILL-226) Remove the deprecated HDFSLocationFactory and related classes

2017-03-27 Thread Terence Yim (JIRA)

Terence Yim created TWILL-226:
-

 Summary: Remove the deprecated HDFSLocationFactory and related 
classes
 Key: TWILL-226
 URL: https://issues.apache.org/jira/browse/TWILL-226
 Project: Apache Twill
  Issue Type: Task
  Components: yarn
Reporter: Terence Yim
Assignee: Terence Yim






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (TWILL-226) Remove the deprecated HDFSLocationFactory and related classes

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-226:
--
Fix Version/s: 0.11.0

> Remove the deprecated HDFSLocationFactory and related classes
> -
>
> Key: TWILL-226
> URL: https://issues.apache.org/jira/browse/TWILL-226
> Project: Apache Twill
>  Issue Type: Task
>  Components: yarn
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Sam William

I was able to make it work with this 1 line change. 

https://github.com/sampd/twill/compare/branch-0.10.0...sampd:v10_test?expand=1 


We have a fat jar with all dependencies included for the application. Could 
that be the cause ?

Sam



> On Mar 27, 2017, at 15:49, Yuliya Feldman  wrote:
> 
> Code of your application you want to be running in YARN I believe :)
> 
> On Mon, Mar 27, 2017 at 3:28 PM, Sam William  wrote:
> 
>> Yes. 22 bytes looks like an empty zip file.  Any idea what should there in
>> the application jar file ?
>> 
>> Sam
>>> On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
>>> 
>>> File is very small - it may be nothing to do with file not found. Either
>>> permissions or something else
>>> 
>>> On Mon, Mar 27, 2017 at 1:17 PM, Sam William 
>> wrote:
>>> 
 I logged into the master host and looked at the nodemanager logs. It
>> fails
 at localizing the application jar.  The files are there in HDFS.  I can
 even see it is able to copy the other files just fine (for example the
 launcher jar and runtime.config)
 
 -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
 /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
 c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
 -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
 /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
 -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
 /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/launcher.
 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
 -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
 /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
 -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
 /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
 -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
 /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
 
 
 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
 nodemanager.containermanager.localizer.LocalizedResource: Resource
 hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
 d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
 appcache/application_1484158548936_11282/filecache/
 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
>> transitioned
 from DOWNLOADING to LOCALIZED
 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
 nodemanager.containermanager.localizer.LocalizedResource: Resource
 hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
 d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
 appcache/application_1484158548936_11282/filecache/
 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
 DOWNLOADING to LOCALIZED
 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.
>> UserGroupInformation:
 PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such
>> file
 or directory
 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
 nodemanager.containermanager.localizer.ResourceLocalizationService: {
 hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
 d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
 bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-
>> 4121-824e-448a12a635c1.jar,
 1490644063924, ARCHIVE, null } failed: No such file or directory
 ENOENT: No such file or directory
   at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
 Method)
   at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
 java:230)
   at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
 RawLocalFileSystem.java:660)
   at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
 DelegateToFileSystem.java:206)
   at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:
>> 251)
   at org.apache.hadoop.fs.FileContext$10.next(
>> FileContext.java:955)
   at org.apache.hadoop.fs.FileContext$10.next(
>> FileContext.java:951)
   at org.apache.hadoop.fs.FSLinkResolver.resolve(
 FSLinkResolver.java:90)
   at org.apache.hadoop.fs.FileContext.setPermission(
 FileContext.java:951)
 
 
> On Mar 27, 2017, at 12:45, Sam William  wrote:
> 
> Hi Terence,
> Im not able to get logs for these jobs. “yarn logs” command does nt
 return anything.
> Sam

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Yuliya Feldman

Code of your application you want to be running in YARN I believe :)

On Mon, Mar 27, 2017 at 3:28 PM, Sam William  wrote:

> Yes. 22 bytes looks like an empty zip file.  Any idea what should there in
> the application jar file ?
>
> Sam
> > On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
> >
> > File is very small - it may be nothing to do with file not found. Either
> > permissions or something else
> >
> > On Mon, Mar 27, 2017 at 1:17 PM, Sam William 
> wrote:
> >
> >> I logged into the master host and looked at the nodemanager logs. It
> fails
> >> at localizing the application jar.  The files are there in HDFS.  I can
> >> even see it is able to copy the other files just fine (for example the
> >> launcher jar and runtime.config)
> >>
> >> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
> >> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
> >> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
> >> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/launcher.
> >> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
> >> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
> >> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
> >> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
> >>
> >>
> >> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
> >> nodemanager.containermanager.localizer.LocalizedResource: Resource
> >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> >> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
> >> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
> >> appcache/application_1484158548936_11282/filecache/
> >> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
> transitioned
> >> from DOWNLOADING to LOCALIZED
> >> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
> >> nodemanager.containermanager.localizer.LocalizedResource: Resource
> >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> >> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
> >> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
> >> appcache/application_1484158548936_11282/filecache/
> >> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
> >> DOWNLOADING to LOCALIZED
> >> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.
> UserGroupInformation:
> >> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such
> file
> >> or directory
> >> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
> >> nodemanager.containermanager.localizer.ResourceLocalizationService: {
> >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> >> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
> >> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-
> 4121-824e-448a12a635c1.jar,
> >> 1490644063924, ARCHIVE, null } failed: No such file or directory
> >> ENOENT: No such file or directory
> >>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
> >> Method)
> >>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
> >> java:230)
> >>at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> >> RawLocalFileSystem.java:660)
> >>at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
> >> DelegateToFileSystem.java:206)
> >>at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:
> 251)
> >>at org.apache.hadoop.fs.FileContext$10.next(
> FileContext.java:955)
> >>at org.apache.hadoop.fs.FileContext$10.next(
> FileContext.java:951)
> >>at org.apache.hadoop.fs.FSLinkResolver.resolve(
> >> FSLinkResolver.java:90)
> >>at org.apache.hadoop.fs.FileContext.setPermission(
> >> FileContext.java:951)
> >>
> >>
> >>> On Mar 27, 2017, at 12:45, Sam William  wrote:
> >>>
> >>> Hi Terence,
> >>>  Im not able to get logs for these jobs. “yarn logs” command does nt
> >> return anything.
> >>> Sam
>  On Mar 26, 2017, at 17:32, Terence Yim  wrote:
> 
>  Hi Sam,
> 
>  I guess it might be related to the missing of the Hadoop conf
> directory
> >> in the container classpath, such that the locationfactory constructed
> from
> >> the container side is not correct. Do you have access to the containers
> >> stdout file? It shows the classpath twill uses.
> 
>  Terence
> >>

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Sam William

Yes. 22 bytes looks like an empty zip file.  Any idea what should there in the 
application jar file ? 

Sam
> On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
> 
> File is very small - it may be nothing to do with file not found. Either
> permissions or something else
> 
> On Mon, Mar 27, 2017 at 1:17 PM, Sam William  wrote:
> 
>> I logged into the master host and looked at the nodemanager logs. It fails
>> at localizing the application jar.  The files are there in HDFS.  I can
>> even see it is able to copy the other files just fine (for example the
>> launcher jar and runtime.config)
>> 
>> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
>> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
>> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
>> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.
>> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
>> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
>> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
>> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
>> 
>> 
>> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.LocalizedResource: Resource
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
>> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
>> appcache/application_1484158548936_11282/filecache/
>> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar) transitioned
>> from DOWNLOADING to LOCALIZED
>> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.LocalizedResource: Resource
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
>> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
>> appcache/application_1484158548936_11282/filecache/
>> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
>> DOWNLOADING to LOCALIZED
>> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such file
>> or directory
>> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.ResourceLocalizationService: {
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
>> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar,
>> 1490644063924, ARCHIVE, null } failed: No such file or directory
>> ENOENT: No such file or directory
>>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
>> Method)
>>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
>> java:230)
>>at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
>> RawLocalFileSystem.java:660)
>>at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
>> DelegateToFileSystem.java:206)
>>at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
>>at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
>>at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
>>at org.apache.hadoop.fs.FSLinkResolver.resolve(
>> FSLinkResolver.java:90)
>>at org.apache.hadoop.fs.FileContext.setPermission(
>> FileContext.java:951)
>> 
>> 
>>> On Mar 27, 2017, at 12:45, Sam William  wrote:
>>> 
>>> Hi Terence,
>>>  Im not able to get logs for these jobs. “yarn logs” command does nt
>> return anything.
>>> Sam
 On Mar 26, 2017, at 17:32, Terence Yim  wrote:
 
 Hi Sam,
 
 I guess it might be related to the missing of the Hadoop conf directory
>> in the container classpath, such that the locationfactory constructed from
>> the container side is not correct. Do you have access to the containers
>> stdout file? It shows the classpath twill uses.
 
 Terence
 
 Sent from my iPhone
 
> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
> 
> It works with Twill-0.9.0. So far I have been able to narrow it down
>> to one commit
> 
> 5986553 (TWILL-63) Speed up application launch time
> 
> Let me see if can nail down to a particular change.
> 
> S

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Yuliya Feldman

File is very small - it may be nothing to do with file not found. Either
permissions or something else

On Mon, Mar 27, 2017 at 1:17 PM, Sam William  wrote:

> I logged into the master host and looked at the nodemanager logs. It fails
> at localizing the application jar.  The files are there in HDFS.  I can
> even see it is able to copy the other files just fine (for example the
> launcher jar and runtime.config)
>
> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.
> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
>
>
> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.localizer.LocalizedResource: Resource
> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
> appcache/application_1484158548936_11282/filecache/
> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar) transitioned
> from DOWNLOADING to LOCALIZED
> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.localizer.LocalizedResource: Resource
> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
> appcache/application_1484158548936_11282/filecache/
> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
> DOWNLOADING to LOCALIZED
> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such file
> or directory
> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
> nodemanager.containermanager.localizer.ResourceLocalizationService: {
> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar,
> 1490644063924, ARCHIVE, null } failed: No such file or directory
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
> Method)
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
> java:230)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> RawLocalFileSystem.java:660)
> at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
> DelegateToFileSystem.java:206)
> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(
> FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.setPermission(
> FileContext.java:951)
>
>
> > On Mar 27, 2017, at 12:45, Sam William  wrote:
> >
> > Hi Terence,
> >   Im not able to get logs for these jobs. “yarn logs” command does nt
> return anything.
> > Sam
> >> On Mar 26, 2017, at 17:32, Terence Yim  wrote:
> >>
> >> Hi Sam,
> >>
> >> I guess it might be related to the missing of the Hadoop conf directory
> in the container classpath, such that the locationfactory constructed from
> the container side is not correct. Do you have access to the containers
> stdout file? It shows the classpath twill uses.
> >>
> >> Terence
> >>
> >> Sent from my iPhone
> >>
> >>> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
> >>>
> >>> It works with Twill-0.9.0. So far I have been able to narrow it down
> to one commit
> >>>
> >>> 5986553 (TWILL-63) Speed up application launch time
> >>>
> >>> Let me see if can nail down to a particular change.
> >>>
> >>> Sam
> >>>
> >>>
>  On Mar 25, 2017, at 13:34, Sam William  wrote:
> 
>  HI Terence,
>  Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages
> for my fat jars.
> 
>  SAm
> > On Mar 25, 2017, at 12:31, Terence Yi

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Sam William

I logged into the master host and looked at the nodemanager logs. It fails at 
localizing the application jar.  The files are there in HDFS.  I can even see 
it is able to copy the other files just fine (for example the launcher jar and 
runtime.config)

-rw-r--r--   3 sam supergroup 22 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
-rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
-rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
-rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
-rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
-rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar


2017-03-27 12:47:45,632 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource 
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/appcache/application_1484158548936_11282/filecache/11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
 transitioned from DOWNLOADING to LOCALIZED
2017-03-27 12:47:45,645 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource 
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/appcache/application_1484158548936_11282/filecache/12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar)
 transitioned from DOWNLOADING to LOCALIZED
2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.UserGroupInformation: 
PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such file or 
directory
2017-03-27 12:47:45,655 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 { 
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar,
 1490644063924, ARCHIVE, null } failed: No such file or directory
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
at 
org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)


> On Mar 27, 2017, at 12:45, Sam William  wrote:
> 
> Hi Terence,
>   Im not able to get logs for these jobs. “yarn logs” command does nt return 
> anything.
> Sam
>> On Mar 26, 2017, at 17:32, Terence Yim  wrote:
>> 
>> Hi Sam,
>> 
>> I guess it might be related to the missing of the Hadoop conf directory in 
>> the container classpath, such that the locationfactory constructed from the 
>> container side is not correct. Do you have access to the containers stdout 
>> file? It shows the classpath twill uses.
>> 
>> Terence
>> 
>> Sent from my iPhone
>> 
>>> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
>>> 
>>> It works with Twill-0.9.0. So far I have been able to narrow it down to one 
>>> commit 
>>> 
>>> 5986553 (TWILL-63) Speed up application launch time
>>> 
>>> Let me see if can nail down to a particular change.
>>> 
>>> Sam
>>> 
>>> 
 On Mar 25, 2017, at 13:34, Sam William  wrote:
 
 HI Terence,
 Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for 
 my fat jars.
 
 SAm
> On Mar 25, 2017, at 12:31, Terence Yim  wrote:
> 
> Hi,
> 
> Haven't seen this error before. What is the version of Hadoop that the 
> cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
> classpath as the FileContext is trying to use local file system instead 
> of the distributed one.
> 
> Terence
>

Re: ENOENT error on upgrading to Twill 0.10.0

2017-03-27 Thread Sam William

Hi Terence,
   Im not able to get logs for these jobs. “yarn logs” command does nt return 
anything.
Sam
> On Mar 26, 2017, at 17:32, Terence Yim  wrote:
> 
> Hi Sam,
> 
> I guess it might be related to the missing of the Hadoop conf directory in 
> the container classpath, such that the locationfactory constructed from the 
> container side is not correct. Do you have access to the containers stdout 
> file? It shows the classpath twill uses.
> 
> Terence
> 
> Sent from my iPhone
> 
>> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
>> 
>> It works with Twill-0.9.0. So far I have been able to narrow it down to one 
>> commit 
>> 
>> 5986553 (TWILL-63) Speed up application launch time
>> 
>> Let me see if can nail down to a particular change.
>> 
>> Sam
>> 
>> 
>>> On Mar 25, 2017, at 13:34, Sam William  wrote:
>>> 
>>> HI Terence,
>>> Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for my 
>>> fat jars.
>>> 
>>> SAm
 On Mar 25, 2017, at 12:31, Terence Yim  wrote:
 
 Hi,
 
 Haven't seen this error before. What is the version of Hadoop that the 
 cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
 classpath as the FileContext is trying to use local file system instead of 
 the distributed one.
 
 Terence
 
 Sent from my iPhone
 
> On Mar 25, 2017, at 12:25 PM, Sam William  wrote:
> 
> Hi,
> I have been using Twill for sometime now and I just tried to upgrade our 
> application from Twill-0.8.0 to 0.10.0.  I havent made any kind of code 
> changes besides changing the Twill version string in the build script.  
> The application fails immediately and I see this on the RM UI. Any idea 
> why this could be happening?
> 
> Diagnostics:
> Application application_1484158548936_11154 failed 2 times due to AM 
> Container for appattempt_1484158548936_11154_02 exited with exitCode: 
> -1000
> For more detailed output, check application tracking page<> Then, click 
> on links to logs of each attempt.
> Diagnostics: No such file or directory
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
> at 
> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at 
> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Failing this attempt. Failing the application.
> 
> 
> Sam
>>> 
>>

[jira] [Resolved] (TWILL-171) Update HDFS delegation token properly for HA mode

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-171.
---
Resolution: Fixed

> Update HDFS delegation token properly for HA mode
> -
>
> Key: TWILL-171
> URL: https://issues.apache.org/jira/browse/TWILL-171
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.7.0-incubating
> Environment: HA-enabled NodeManager, Kerberos-enabled
>Reporter: Ali Anwar
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> There is an issue https://issues.apache.org/jira/browse/HDFS-9276, which can 
> be worked around to update HDFS delegation token properly for HA mode.
> See: https://issues.cask.co/browse/CDAP-5844



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (TWILL-194) KMS delegation token is not included if the FileContextLocationFactory is ued

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-194.
---
Resolution: Fixed

> KMS delegation token is not included if the FileContextLocationFactory is ued
> -
>
> Key: TWILL-194
> URL: https://issues.apache.org/jira/browse/TWILL-194
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.7.0-incubating, 0.8.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> It is caused by HDFS-10296 and Twill can provide a workaround for that by 
> acquiring the KMS delegation token explicitly when the 
> FileContextLocationFactory is used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #42: (TWILL-171) Clone the HDFS delegation in HA mode.

2017-03-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/42


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-171) Update HDFS delegation token properly for HA mode

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943918#comment-15943918
 ] 

ASF GitHub Bot commented on TWILL-171:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/42


> Update HDFS delegation token properly for HA mode
> -
>
> Key: TWILL-171
> URL: https://issues.apache.org/jira/browse/TWILL-171
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.7.0-incubating
> Environment: HA-enabled NodeManager, Kerberos-enabled
>Reporter: Ali Anwar
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> There is an issue https://issues.apache.org/jira/browse/HDFS-9276, which can 
> be worked around to update HDFS delegation token properly for HA mode.
> See: https://issues.cask.co/browse/CDAP-5844



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-194) KMS delegation token is not included if the FileContextLocationFactory is ued

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943919#comment-15943919
 ] 

ASF GitHub Bot commented on TWILL-194:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/43


> KMS delegation token is not included if the FileContextLocationFactory is ued
> -
>
> Key: TWILL-194
> URL: https://issues.apache.org/jira/browse/TWILL-194
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.7.0-incubating, 0.8.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> It is caused by HDFS-10296 and Twill can provide a workaround for that by 
> acquiring the KMS delegation token explicitly when the 
> FileContextLocationFactory is used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #43: (TWILL-194) Acquires KMS delegation token correctly

2017-03-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/43


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-194) KMS delegation token is not included if the FileContextLocationFactory is ued

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943906#comment-15943906
 ] 

ASF GitHub Bot commented on TWILL-194:
--

Github user anew commented on the issue:

https://github.com/apache/twill/pull/43
  
LGTM


> KMS delegation token is not included if the FileContextLocationFactory is ued
> -
>
> Key: TWILL-194
> URL: https://issues.apache.org/jira/browse/TWILL-194
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.7.0-incubating, 0.8.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> It is caused by HDFS-10296 and Twill can provide a workaround for that by 
> acquiring the KMS delegation token explicitly when the 
> FileContextLocationFactory is used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill issue #43: (TWILL-194) Acquires KMS delegation token correctly

2017-03-27 Thread anew

Github user anew commented on the issue:

https://github.com/apache/twill/pull/43
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-171) Update HDFS delegation token properly for HA mode

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943878#comment-15943878
 ] 

ASF GitHub Bot commented on TWILL-171:
--

Github user anew commented on the issue:

https://github.com/apache/twill/pull/42
  
IMO this is the cleaner, if verbose, way. LGTM +1


> Update HDFS delegation token properly for HA mode
> -
>
> Key: TWILL-171
> URL: https://issues.apache.org/jira/browse/TWILL-171
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.7.0-incubating
> Environment: HA-enabled NodeManager, Kerberos-enabled
>Reporter: Ali Anwar
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> There is an issue https://issues.apache.org/jira/browse/HDFS-9276, which can 
> be worked around to update HDFS delegation token properly for HA mode.
> See: https://issues.cask.co/browse/CDAP-5844



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill issue #42: (TWILL-171) Clone the HDFS delegation in HA mode.

2017-03-27 Thread anew

Github user anew commented on the issue:

https://github.com/apache/twill/pull/42
  
IMO this is the cleaner, if verbose, way. LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill pull request #41: Add website link to readme

2017-03-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/41


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill issue #41: Add website link to readme

2017-03-27 Thread chtyim

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/41
  
LGTM. Merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (TWILL-194) KMS delegation token is not included if the FileContextLocationFactory is ued

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim updated TWILL-194:
--
Fix Version/s: 0.11.0

> KMS delegation token is not included if the FileContextLocationFactory is ued
> -
>
> Key: TWILL-194
> URL: https://issues.apache.org/jira/browse/TWILL-194
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.7.0-incubating, 0.8.0
>Reporter: Terence Yim
> Fix For: 0.11.0
>
>
> It is caused by HDFS-10296 and Twill can provide a workaround for that by 
> acquiring the KMS delegation token explicitly when the 
> FileContextLocationFactory is used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (TWILL-194) KMS delegation token is not included if the FileContextLocationFactory is ued

2017-03-27 Thread Terence Yim (JIRA)


 [ 
https://issues.apache.org/jira/browse/TWILL-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim reassigned TWILL-194:
-

Assignee: Terence Yim

> KMS delegation token is not included if the FileContextLocationFactory is ued
> -
>
> Key: TWILL-194
> URL: https://issues.apache.org/jira/browse/TWILL-194
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.7.0-incubating, 0.8.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> It is caused by HDFS-10296 and Twill can provide a workaround for that by 
> acquiring the KMS delegation token explicitly when the 
> FileContextLocationFactory is used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-171) Update HDFS delegation token properly for HA mode

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943748#comment-15943748
 ] 

ASF GitHub Bot commented on TWILL-171:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/42
  
@serranom True. We can also try to get it via the `LocationFactory` from 
the application location, however it won't really make the code easier to 
understand (there would be unwrapping and `instanceof` check)


> Update HDFS delegation token properly for HA mode
> -
>
> Key: TWILL-171
> URL: https://issues.apache.org/jira/browse/TWILL-171
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.7.0-incubating
> Environment: HA-enabled NodeManager, Kerberos-enabled
>Reporter: Ali Anwar
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> There is an issue https://issues.apache.org/jira/browse/HDFS-9276, which can 
> be worked around to update HDFS delegation token properly for HA mode.
> See: https://issues.cask.co/browse/CDAP-5844



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill issue #42: (TWILL-171) Clone the HDFS delegation in HA mode.

2017-03-27 Thread chtyim

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/42
  
@serranom True. We can also try to get it via the `LocationFactory` from 
the application location, however it won't really make the code easier to 
understand (there would be unwrapping and `instanceof` check)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-194) KMS delegation token is not included if the FileContextLocationFactory is ued

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943744#comment-15943744
 ] 

ASF GitHub Bot commented on TWILL-194:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/43

(TWILL-194) Acquires KMS delegation token correctly

- This is to workaround HDFS-10296 that if FileContext object is used
  to acquire delegation token, the KMS delegation token is not included

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-194

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/43.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #43


commit d37d8af52803979acec0704e6324a8d7fe22da2c
Author: Terence Yim 
Date:   2017-03-27T17:58:54Z

(TWILL-194) Acquires KMS delegation token correctly

- This is to workaround HDFS-10296 that if FileContext object is used
  to acquire delegation token, the KMS delegation token is not included




> KMS delegation token is not included if the FileContextLocationFactory is ued
> -
>
> Key: TWILL-194
> URL: https://issues.apache.org/jira/browse/TWILL-194
> Project: Apache Twill
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 0.7.0-incubating, 0.8.0
>Reporter: Terence Yim
>
> It is caused by HDFS-10296 and Twill can provide a workaround for that by 
> acquiring the KMS delegation token explicitly when the 
> FileContextLocationFactory is used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #43: (TWILL-194) Acquires KMS delegation token correctly

2017-03-27 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/43

(TWILL-194) Acquires KMS delegation token correctly

- This is to workaround HDFS-10296 that if FileContext object is used
  to acquire delegation token, the KMS delegation token is not included

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-194

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/43.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #43


commit d37d8af52803979acec0704e6324a8d7fe22da2c
Author: Terence Yim 
Date:   2017-03-27T17:58:54Z

(TWILL-194) Acquires KMS delegation token correctly

- This is to workaround HDFS-10296 that if FileContext object is used
  to acquire delegation token, the KMS delegation token is not included




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-171) Update HDFS delegation token properly for HA mode

2017-03-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/TWILL-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943738#comment-15943738
 ] 

ASF GitHub Bot commented on TWILL-171:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/42

(TWILL-171) Clone the HDFS delegation in HA mode.

- This is for working around HDFS-9276

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/42.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #42


commit 2813b85f3cfa5ad0e37fd68cf06b1c4d577db76f
Author: Terence Yim 
Date:   2017-03-27T17:51:26Z

(TWILL-171) Clone the HDFS delegation in HA mode.

- This is for working around HDFS-9276




> Update HDFS delegation token properly for HA mode
> -
>
> Key: TWILL-171
> URL: https://issues.apache.org/jira/browse/TWILL-171
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.7.0-incubating
> Environment: HA-enabled NodeManager, Kerberos-enabled
>Reporter: Ali Anwar
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> There is an issue https://issues.apache.org/jira/browse/HDFS-9276, which can 
> be worked around to update HDFS delegation token properly for HA mode.
> See: https://issues.cask.co/browse/CDAP-5844



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #42: (TWILL-171) Clone the HDFS delegation in HA mode.

2017-03-27 Thread chtyim

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/42

(TWILL-171) Clone the HDFS delegation in HA mode.

- This is for working around HDFS-9276

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/42.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #42


commit 2813b85f3cfa5ad0e37fd68cf06b1c4d577db76f
Author: Terence Yim 
Date:   2017-03-27T17:51:26Z

(TWILL-171) Clone the HDFS delegation in HA mode.

- This is for working around HDFS-9276




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] twill pull request #41: Add website link to readme

2017-03-27 Thread keith-turner

GitHub user keith-turner opened a pull request:

https://github.com/apache/twill/pull/41

Add website link to readme



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/keith-turner/twill link

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/41.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #41


commit 46fa6a65517f2fb5f5c667b832f865565425ae81
Author: Keith Turner 
Date:   2017-03-27T15:48:08Z

Add website link to readme




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

46 matches

Mail list logo