Re: ENOENT error on upgrading to Twill 0.10.0

[Yuliya Feldman]

Code of your application you want to be running in YARN I believe :)

On Mon, Mar 27, 2017 at 3:28 PM, Sam William  wrote:

> Yes. 22 bytes looks like an empty zip file.  Any idea what should there in
> the application jar file ?
>
> Sam
> > On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
> >
> > File is very small - it may be nothing to do with file not found. Either
> > permissions or something else
> >
> > On Mon, Mar 27, 2017 at 1:17 PM, Sam William 
> wrote:
> >
> >> I logged into the master host and looked at the nodemanager logs. It
> fails
> >> at localizing the application jar.  The files are there in HDFS.  I can
> >> even see it is able to copy the other files just fine (for example the
> >> launcher jar and runtime.config)
> >>
> >> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
> >> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
> >> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
> >> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> 44a506886fc1/launcher.
> >> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
> >> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
> >> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
> >> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
> >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
> >> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
> >>
> >>
> >> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
> >> nodemanager.containermanager.localizer.LocalizedResource: Resource
> >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> >> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
> >> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
> >> appcache/application_1484158548936_11282/filecache/
> >> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
> transitioned
> >> from DOWNLOADING to LOCALIZED
> >> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
> >> nodemanager.containermanager.localizer.LocalizedResource: Resource
> >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> >> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
> >> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
> >> appcache/application_1484158548936_11282/filecache/
> >> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
> >> DOWNLOADING to LOCALIZED
> >> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.
> UserGroupInformation:
> >> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such
> file
> >> or directory
> >> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
> >> nodemanager.containermanager.localizer.ResourceLocalizationService: {
> >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
> >> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
> >> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-
> 4121-824e-448a12a635c1.jar,
> >> 1490644063924, ARCHIVE, null } failed: No such file or directory
> >> ENOENT: No such file or directory
> >>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
> >> Method)
> >>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
> >> java:230)
> >>at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> >> RawLocalFileSystem.java:660)
> >>at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
> >> DelegateToFileSystem.java:206)
> >>at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:
> 251)
> >>at org.apache.hadoop.fs.FileContext$10.next(
> FileContext.java:955)
> >>at org.apache.hadoop.fs.FileContext$10.next(
> FileContext.java:951)
> >>at org.apache.hadoop.fs.FSLinkResolver.resolve(
> >> FSLinkResolver.java:90)
> >>at org.apache.hadoop.fs.FileContext.setPermission(
> >> FileContext.java:951)
> >>
> >>
> >>> On Mar 27, 2017, at 12:45, Sam William  wrote:
> >>>
> >>> Hi Terence,
> >>>  Im not able to get logs for these jobs. “yarn logs” command does nt
> >> return anything.
> >>> Sam
>  On Mar 26, 2017, at 17:32, Terence Yim  wrote:
> 
>  Hi Sam,
> 
>  I guess it might be related to the missing of the Hadoop conf
> directory
> >> in the container classpath, such that the locationfactory constructed
> from
> >> the container side is not correct. Do you have 

Re: ENOENT error on upgrading to Twill 0.10.0

[Sam William]

Yes. 22 bytes looks like an empty zip file.  Any idea what should there in the 
application jar file ? 

Sam
> On Mar 27, 2017, at 13:22, Yuliya Feldman  wrote:
> 
> File is very small - it may be nothing to do with file not found. Either
> permissions or something else
> 
> On Mon, Mar 27, 2017 at 1:17 PM, Sam William  wrote:
> 
>> I logged into the master host and looked at the nodemanager logs. It fails
>> at localizing the application jar.  The files are there in HDFS.  I can
>> even see it is able to copy the other files just fine (for example the
>> launcher jar and runtime.config)
>> 
>> -rw-r--r--   3 sam supergroup 22 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece
>> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
>> -rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
>> -rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.
>> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
>> -rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
>> -rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
>> -rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47
>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-
>> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
>> 
>> 
>> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.LocalizedResource: Resource
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-
>> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/
>> appcache/application_1484158548936_11282/filecache/
>> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar) transitioned
>> from DOWNLOADING to LOCALIZED
>> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.LocalizedResource: Resource
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-
>> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/
>> appcache/application_1484158548936_11282/filecache/
>> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from
>> DOWNLOADING to LOCALIZED
>> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such file
>> or directory
>> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server.
>> nodemanager.containermanager.localizer.ResourceLocalizationService: {
>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-
>> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-
>> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar,
>> 1490644063924, ARCHIVE, null } failed: No such file or directory
>> ENOENT: No such file or directory
>>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
>> Method)
>>at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.
>> java:230)
>>at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
>> RawLocalFileSystem.java:660)
>>at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(
>> DelegateToFileSystem.java:206)
>>at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
>>at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
>>at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
>>at org.apache.hadoop.fs.FSLinkResolver.resolve(
>> FSLinkResolver.java:90)
>>at org.apache.hadoop.fs.FileContext.setPermission(
>> FileContext.java:951)
>> 
>> 
>>> On Mar 27, 2017, at 12:45, Sam William  wrote:
>>> 
>>> Hi Terence,
>>>  Im not able to get logs for these jobs. “yarn logs” command does nt
>> return anything.
>>> Sam
 On Mar 26, 2017, at 17:32, Terence Yim  wrote:
 
 Hi Sam,
 
 I guess it might be related to the missing of the Hadoop conf directory
>> in the container classpath, such that the locationfactory constructed from
>> the container side is not correct. Do you have access to the containers
>> stdout file? It shows the classpath twill uses.
 
 Terence
 
 Sent from my iPhone
 
> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
> 
> It works with Twill-0.9.0. So far I have been able to narrow it down
>> to one commit
> 
> 5986553 (TWILL-63) Speed up 

Re: ENOENT error on upgrading to Twill 0.10.0

[Sam William]

I logged into the master host and looked at the nodemanager logs. It fails at 
localizing the application jar.  The files are there in HDFS.  I can even see 
it is able to copy the other files just fine (for example the launcher jar and 
runtime.config)

-rw-r--r--   3 sam supergroup 22 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
-rw-r--r--   3 sam supergroup5991970 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
-rw-r--r--   3 sam supergroup   5725 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
-rw-r--r--   3 sam supergroup   1038 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
-rw-r--r--   3 sam supergroup   2072 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
-rw-r--r--   3 sam supergroup   48245414 2017-03-27 12:47 
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar


2017-03-27 12:47:45,632 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource 
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/appcache/application_1484158548936_11282/filecache/11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
 transitioned from DOWNLOADING to LOCALIZED
2017-03-27 12:47:45,645 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource 
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/appcache/application_1484158548936_11282/filecache/12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar)
 transitioned from DOWNLOADING to LOCALIZED
2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.UserGroupInformation: 
PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such file or 
directory
2017-03-27 12:47:45,655 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 { 
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar,
 1490644063924, ARCHIVE, null } failed: No such file or directory
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
at 
org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)


> On Mar 27, 2017, at 12:45, Sam William  wrote:
> 
> Hi Terence,
>   Im not able to get logs for these jobs. “yarn logs” command does nt return 
> anything.
> Sam
>> On Mar 26, 2017, at 17:32, Terence Yim  wrote:
>> 
>> Hi Sam,
>> 
>> I guess it might be related to the missing of the Hadoop conf directory in 
>> the container classpath, such that the locationfactory constructed from the 
>> container side is not correct. Do you have access to the containers stdout 
>> file? It shows the classpath twill uses.
>> 
>> Terence
>> 
>> Sent from my iPhone
>> 
>>> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
>>> 
>>> It works with Twill-0.9.0. So far I have been able to narrow it down to one 
>>> commit 
>>> 
>>> 5986553 (TWILL-63) Speed up application launch time
>>> 
>>> Let me see if can nail down to a particular change.
>>> 
>>> Sam
>>> 
>>> 
 On Mar 25, 2017, at 13:34, Sam William  wrote:
 
 HI Terence,
 Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for 
 my fat jars.
 
 SAm
> On Mar 25, 2017, at 12:31, Terence Yim  wrote:
> 
> Hi,
> 
> Haven't seen this error before. What is the version of Hadoop that the 
> cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
> classpath as the FileContext 

Re: ENOENT error on upgrading to Twill 0.10.0

[Sam William]

Hi Terence,
   Im not able to get logs for these jobs. “yarn logs” command does nt return 
anything.
Sam
> On Mar 26, 2017, at 17:32, Terence Yim  wrote:
> 
> Hi Sam,
> 
> I guess it might be related to the missing of the Hadoop conf directory in 
> the container classpath, such that the locationfactory constructed from the 
> container side is not correct. Do you have access to the containers stdout 
> file? It shows the classpath twill uses.
> 
> Terence
> 
> Sent from my iPhone
> 
>> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
>> 
>> It works with Twill-0.9.0. So far I have been able to narrow it down to one 
>> commit 
>> 
>> 5986553 (TWILL-63) Speed up application launch time
>> 
>> Let me see if can nail down to a particular change.
>> 
>> Sam
>> 
>> 
>>> On Mar 25, 2017, at 13:34, Sam William  wrote:
>>> 
>>> HI Terence,
>>> Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for my 
>>> fat jars.
>>> 
>>> SAm
 On Mar 25, 2017, at 12:31, Terence Yim  wrote:
 
 Hi,
 
 Haven't seen this error before. What is the version of Hadoop that the 
 cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
 classpath as the FileContext is trying to use local file system instead of 
 the distributed one.
 
 Terence
 
 Sent from my iPhone
 
> On Mar 25, 2017, at 12:25 PM, Sam William  wrote:
> 
> Hi,
> I have been using Twill for sometime now and I just tried to upgrade our 
> application from Twill-0.8.0 to 0.10.0.  I havent made any kind of code 
> changes besides changing the Twill version string in the build script.  
> The application fails immediately and I see this on the RM UI. Any idea 
> why this could be happening?
> 
> Diagnostics:
> Application application_1484158548936_11154 failed 2 times due to AM 
> Container for appattempt_1484158548936_11154_02 exited with exitCode: 
> -1000
> For more detailed output, check application tracking page<> Then, click 
> on links to logs of each attempt.
> Diagnostics: No such file or directory
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
> at 
> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at 
> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Failing this attempt. Failing the application.
> 
> 
> Sam
>>> 
>> 

Re: ENOENT error on upgrading to Twill 0.10.0

[Terence Yim]

Hi Sam,
 
I guess it might be related to the missing of the Hadoop conf directory in the 
container classpath, such that the locationfactory constructed from the 
container side is not correct. Do you have access to the containers stdout 
file? It shows the classpath twill uses.

Terence

Sent from my iPhone

> On Mar 26, 2017, at 3:16 PM, Sam William  wrote:
> 
> It works with Twill-0.9.0. So far I have been able to narrow it down to one 
> commit 
> 
> 5986553 (TWILL-63) Speed up application launch time
> 
> Let me see if can nail down to a particular change.
> 
> Sam
> 
> 
>> On Mar 25, 2017, at 13:34, Sam William  wrote:
>> 
>> HI Terence,
>> Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for my 
>> fat jars.
>> 
>> SAm
>>> On Mar 25, 2017, at 12:31, Terence Yim  wrote:
>>> 
>>> Hi,
>>> 
>>> Haven't seen this error before. What is the version of Hadoop that the 
>>> cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
>>> classpath as the FileContext is trying to use local file system instead of 
>>> the distributed one.
>>> 
>>> Terence
>>> 
>>> Sent from my iPhone
>>> 
 On Mar 25, 2017, at 12:25 PM, Sam William  wrote:
 
 Hi,
 I have been using Twill for sometime now and I just tried to upgrade our 
 application from Twill-0.8.0 to 0.10.0.  I havent made any kind of code 
 changes besides changing the Twill version string in the build script.  
 The application fails immediately and I see this on the RM UI. Any idea 
 why this could be happening?
 
 Diagnostics:
 Application application_1484158548936_11154 failed 2 times due to AM 
 Container for appattempt_1484158548936_11154_02 exited with exitCode: 
 -1000
 For more detailed output, check application tracking page<> Then, click on 
 links to logs of each attempt.
 Diagnostics: No such file or directory
 ENOENT: No such file or directory
 at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
 at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
 at 
 org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
 at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
 at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
 at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
 at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
 at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
 at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
 at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
 at 
 org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
 at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
 at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Failing this attempt. Failing the application.
 
 
 Sam
>> 
> 

Re: ENOENT error on upgrading to Twill 0.10.0

[Sam William]

It works with Twill-0.9.0. So far I have been able to narrow it down to one 
commit 

5986553 (TWILL-63) Speed up application launch time

Let me see if can nail down to a particular change.

Sam


> On Mar 25, 2017, at 13:34, Sam William  wrote:
> 
> HI Terence,
>  Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for my 
> fat jars.
> 
> SAm
>> On Mar 25, 2017, at 12:31, Terence Yim  wrote:
>> 
>> Hi,
>> 
>> Haven't seen this error before. What is the version of Hadoop that the 
>> cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
>> classpath as the FileContext is trying to use local file system instead of 
>> the distributed one.
>> 
>> Terence
>> 
>> Sent from my iPhone
>> 
>>> On Mar 25, 2017, at 12:25 PM, Sam William  wrote:
>>> 
>>> Hi,
>>> I have been using Twill for sometime now and I just tried to upgrade our 
>>> application from Twill-0.8.0 to 0.10.0.  I havent made any kind of code 
>>> changes besides changing the Twill version string in the build script.  The 
>>> application fails immediately and I see this on the RM UI. Any idea why 
>>> this could be happening?
>>> 
>>> Diagnostics:
>>> Application application_1484158548936_11154 failed 2 times due to AM 
>>> Container for appattempt_1484158548936_11154_02 exited with exitCode: 
>>> -1000
>>> For more detailed output, check application tracking page<> Then, click on 
>>> links to logs of each attempt.
>>> Diagnostics: No such file or directory
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
>>> at 
>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
>>> at 
>>> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
>>> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
>>> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>>> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>> at 
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>>> at 
>>> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Failing this attempt. Failing the application.
>>> 
>>> 
>>> Sam
> 

Re: ENOENT error on upgrading to Twill 0.10.0

[Sam William]

HI Terence,
  Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for my 
fat jars.

SAm
> On Mar 25, 2017, at 12:31, Terence Yim  wrote:
> 
> Hi,
> 
> Haven't seen this error before. What is the version of Hadoop that the 
> cluster is running with? Also, seems like the $HADOOP_CONF is not in the 
> classpath as the FileContext is trying to use local file system instead of 
> the distributed one.
> 
> Terence
> 
> Sent from my iPhone
> 
>> On Mar 25, 2017, at 12:25 PM, Sam William  wrote:
>> 
>> Hi,
>> I have been using Twill for sometime now and I just tried to upgrade our 
>> application from Twill-0.8.0 to 0.10.0.  I havent made any kind of code 
>> changes besides changing the Twill version string in the build script.  The 
>> application fails immediately and I see this on the RM UI. Any idea why this 
>> could be happening?
>> 
>> Diagnostics:
>> Application application_1484158548936_11154 failed 2 times due to AM 
>> Container for appattempt_1484158548936_11154_02 exited with exitCode: 
>> -1000
>> For more detailed output, check application tracking page<> Then, click on 
>> links to logs of each attempt.
>> Diagnostics: No such file or directory
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
>> at 
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
>> at 
>> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
>> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
>> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>> at 
>> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Failing this attempt. Failing the application.
>> 
>> 
>> Sam

Re: ENOENT error on upgrading to Twill 0.10.0

[Terence Yim]

Hi,

Haven't seen this error before. What is the version of Hadoop that the cluster 
is running with? Also, seems like the $HADOOP_CONF is not in the classpath as 
the FileContext is trying to use local file system instead of the distributed 
one.

Terence

Sent from my iPhone

> On Mar 25, 2017, at 12:25 PM, Sam William  wrote:
> 
> Hi,
>  I have been using Twill for sometime now and I just tried to upgrade our 
> application from Twill-0.8.0 to 0.10.0.  I havent made any kind of code 
> changes besides changing the Twill version string in the build script.  The 
> application fails immediately and I see this on the RM UI. Any idea why this 
> could be happening?
> 
> Diagnostics:
> Application application_1484158548936_11154 failed 2 times due to AM 
> Container for appattempt_1484158548936_11154_02 exited with exitCode: 
> -1000
> For more detailed output, check application tracking page<> Then, click on 
> links to logs of each attempt.
> Diagnostics: No such file or directory
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
> at 
> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at 
> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Failing this attempt. Failing the application.
> 
> 
> Sam