[jira] [Resolved] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

[Terence Yim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Terence Yim resolved TWILL-227.
---
Resolution: Fixed

> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946194#comment-15946194
 ] 

ASF GitHub Bot commented on TWILL-227:
--

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/46


> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #46: (TWILL-227) Disabling caching of FileSystem instance...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/twill/pull/46


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: ENOENT error on upgrading to Twill 0.10.0

[Sam William]

Terence,
  Yes the twill classes are included in the fat jar.  I ll file a JIRA soon.

Thanks,
Sam
> On Mar 27, 2017, at 20:39, Terence Yim  wrote:
> 
> Hi Sam,
> 
> Do you have twill classes included in the fat jar as well? I think it might 
> be affecting the application jar that twill generate. Would you mind filing a 
> JIRA on this?
> 
> Thanks,
> Terence
> 
>> On Mar 27, 2017, at 3:55 PM, Sam William  wrote:
>> 
>> I was able to make it work with this 1 line change. 
>> 
>> https://github.com/sampd/twill/compare/branch-0.10.0...sampd:v10_test?expand=1
>>  
>> 
>> 
>> We have a fat jar with all dependencies included for the application. Could 
>> that be the cause ?
>> 
>> Sam
>> 
>> 
>> 
>>> On Mar 27, 2017, at 15:49, Yuliya Feldman  wrote:
>>> 
>>> Code of your application you want to be running in YARN I believe :)
>>> 
>>> On Mon, Mar 27, 2017 at 3:28 PM, Sam William  wrote:
>>> 
 Yes. 22 bytes looks like an empty zip file.  Any idea what should there in
 the application jar file ?
 
 Sam
> On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
> 
> File is very small - it may be nothing to do with file not found. Either
> permissions or something else
> 
> On Mon, Mar 27, 2017 at 1:17 PM, Sam William 
 wrote:
> 
>> I logged into the master host and looked at the nodemanager logs. It
 fails
>> at localizing the application jar.  The files are there in HDFS.  I can
>> even see it is able to copy the other files just fine (for example the
>> launcher jar and runtime.config)
>> 
>> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
>> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
>> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
>> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
 44a506886fc1/launcher.
>> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
>> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
>> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
>> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
>> 
>> 
>> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.LocalizedResource: Resource
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
>> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
>> appcache/application_1484158548936_11282/filecache/
>> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
 transitioned
>> from DOWNLOADING to LOCALIZED
>> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.LocalizedResource: Resource
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
>> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
>> appcache/application_1484158548936_11282/filecache/
>> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
>> DOWNLOADING to LOCALIZED
>> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.
 UserGroupInformation:
>> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such
 file
>> or directory
>> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.ResourceLocalizationService: {
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
>> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-
 4121-824e-448a12a635c1.jar,
>> 1490644063924, ARCHIVE, null } failed: No such file or directory
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
>> Method)
>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
>> java:230)
>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
>> RawLocalFileSystem.java:660)
>> at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
>> DelegateToFileSystem.java:206

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946000#comment-15946000
 ] 

ASF GitHub Bot commented on TWILL-227:
--

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/46#discussion_r108542463
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -159,19 +159,19 @@ public static ApplicationId createApplicationId(long 
timestamp, int id) {
   return ImmutableList.of();
 }
 
-FileSystem fileSystem = getFileSystem(locationFactory, config);
-
-if (fileSystem == null) {
-  LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
-  return ImmutableList.of();
-}
+try (FileSystem fileSystem = getFileSystem(locationFactory)) {
+  if (fileSystem == null) {
+LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
--- End diff --

Updated


> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #46: (TWILL-227) Disabling caching of FileSystem instance...

[chtyim]

Github user chtyim commented on a diff in the pull request:

https://github.com/apache/twill/pull/46#discussion_r108542463
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -159,19 +159,19 @@ public static ApplicationId createApplicationId(long 
timestamp, int id) {
   return ImmutableList.of();
 }
 
-FileSystem fileSystem = getFileSystem(locationFactory, config);
-
-if (fileSystem == null) {
-  LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
-  return ImmutableList.of();
-}
+try (FileSystem fileSystem = getFileSystem(locationFactory)) {
+  if (fileSystem == null) {
+LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
--- End diff --

Updated


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-227) Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is called with different user

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945888#comment-15945888
 ] 

ASF GitHub Bot commented on TWILL-227:
--

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/46#discussion_r108490923
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -159,19 +159,19 @@ public static ApplicationId createApplicationId(long 
timestamp, int id) {
   return ImmutableList.of();
 }
 
-FileSystem fileSystem = getFileSystem(locationFactory, config);
-
-if (fileSystem == null) {
-  LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
-  return ImmutableList.of();
-}
+try (FileSystem fileSystem = getFileSystem(locationFactory)) {
+  if (fileSystem == null) {
+LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
--- End diff --

seems that this message should be logged inside getFileSystem()


> Possible leakage of FileSystem object when YarnUtils.addDelegationTokens is 
> called with different user
> --
>
> Key: TWILL-227
> URL: https://issues.apache.org/jira/browse/TWILL-227
> Project: Apache Twill
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileSystem.get}} caches instance for different UGI by default and the 
> cache will never cleanup until process shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #46: (TWILL-227) Disabling caching of FileSystem instance...

[anew]

Github user anew commented on a diff in the pull request:

https://github.com/apache/twill/pull/46#discussion_r108490923
  
--- Diff: 
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -159,19 +159,19 @@ public static ApplicationId createApplicationId(long 
timestamp, int id) {
   return ImmutableList.of();
 }
 
-FileSystem fileSystem = getFileSystem(locationFactory, config);
-
-if (fileSystem == null) {
-  LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
-  return ImmutableList.of();
-}
+try (FileSystem fileSystem = getFileSystem(locationFactory)) {
+  if (fileSystem == null) {
+LOG.warn("Unexpected: LocationFactory is not backed by 
FileContextLocationFactory");
--- End diff --

seems that this message should be logged inside getFileSystem()


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-223) FileContextLocationFactory should use FileContext instance based on the caller UGI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945816#comment-15945816
 ] 

ASF GitHub Bot commented on TWILL-223:
--

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/47

(TWILL-223) Make FileContextLocationFactory UGI aware

- Use different FileContext object based on the caller UGI
- Allows sharing the same factory instance for different user

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-223

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/47.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #47


commit 3472cf55c9a7de912997c4e8f567a1192f8f8823
Author: Terence Yim 
Date:   2017-03-28T19:53:43Z

(TWILL-223) Make FileContextLocationFactory UGI aware

- Use different FileContext object based on the caller UGI
- Allows sharing the same factory instance for different user




> FileContextLocationFactory should use FileContext instance based on the 
> caller UGI
> --
>
> Key: TWILL-223
> URL: https://issues.apache.org/jira/browse/TWILL-223
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileContextLocationFactory}} internally has a cached {{FileContext}} 
> object that was created when the factory was created. However, when the 
> {{getFileContext}} is called (directly or via one of those {{create}} 
> methods), the UGI might be different then the cached one. If the cached one 
> is returning, FS operations will be performed with the user who creates the 
> factory, not the one who calls the {{create}} method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] twill pull request #47: (TWILL-223) Make FileContextLocationFactory UGI awar...

[chtyim]

GitHub user chtyim opened a pull request:

https://github.com/apache/twill/pull/47

(TWILL-223) Make FileContextLocationFactory UGI aware

- Use different FileContext object based on the caller UGI
- Allows sharing the same factory instance for different user

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chtyim/twill feature/TWILL-223

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/47.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #47


commit 3472cf55c9a7de912997c4e8f567a1192f8f8823
Author: Terence Yim 
Date:   2017-03-28T19:53:43Z

(TWILL-223) Make FileContextLocationFactory UGI aware

- Use different FileContext object based on the caller UGI
- Allows sharing the same factory instance for different user




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (TWILL-223) FileContextLocationFactory should use FileContext instance based on the caller UGI

[Terence Yim (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945810#comment-15945810
 ] 

Terence Yim commented on TWILL-223:
---

With this enhancement, the same {{FileContextLocationFactory}} can be reused 
for different user, hence allowing submitting application with different UGI 
via the same {{TwillRunner}}.

> FileContextLocationFactory should use FileContext instance based on the 
> caller UGI
> --
>
> Key: TWILL-223
> URL: https://issues.apache.org/jira/browse/TWILL-223
> Project: Apache Twill
>  Issue Type: Bug
>Reporter: Terence Yim
>Assignee: Terence Yim
> Fix For: 0.11.0
>
>
> The {{FileContextLocationFactory}} internally has a cached {{FileContext}} 
> object that was created when the factory was created. However, when the 
> {{getFileContext}} is called (directly or via one of those {{create}} 
> methods), the UGI might be different then the cached one. If the cached one 
> is returning, FS operations will be performed with the user who creates the 
> factory, not the one who calls the {{create}} method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)