I logged into the master host and looked at the nodemanager logs. It fails at
localizing the application jar. The files are there in HDFS. I can even see
it is able to copy the other files just fine (for example the launcher jar and
runtime.config)
-rw-r--r-- 3 sam supergroup 22 2017-03-27 12:47
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar
-rw-r--r-- 3 sam supergroup 5991970 2017-03-27 12:47
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d.
-rw-r--r-- 3 sam supergroup 5725 2017-03-27 12:47
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar
-rw-r--r-- 3 sam supergroup 1038 2017-03-27 12:47
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json
-rw-r--r-- 3 sam supergroup 2072 2017-03-27 12:47
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar
-rw-r--r-- 3 sam supergroup 48245414 2017-03-27 12:47
/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar
2017-03-27 12:47:45,632 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/appcache/application_1484158548936_11282/filecache/11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar)
transitioned from DOWNLOADING to LOCALIZED
2017-03-27 12:47:45,645 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/appcache/application_1484158548936_11282/filecache/12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar)
transitioned from DOWNLOADING to LOCALIZED
2017-03-27 12:47:45,651 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such file or
directory
2017-03-27 12:47:45,655 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
{
hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384-44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar,
1490644063924, ARCHIVE, null } failed: No such file or directory
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
at
org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
> On Mar 27, 2017, at 12:45, Sam William <[email protected]> wrote:
>
> Hi Terence,
> Im not able to get logs for these jobs. “yarn logs” command does nt return
> anything.
> Sam
>> On Mar 26, 2017, at 17:32, Terence Yim <[email protected]> wrote:
>>
>> Hi Sam,
>>
>> I guess it might be related to the missing of the Hadoop conf directory in
>> the container classpath, such that the locationfactory constructed from the
>> container side is not correct. Do you have access to the containers stdout
>> file? It shows the classpath twill uses.
>>
>> Terence
>>
>> Sent from my iPhone
>>
>>> On Mar 26, 2017, at 3:16 PM, Sam William <[email protected]> wrote:
>>>
>>> It works with Twill-0.9.0. So far I have been able to narrow it down to one
>>> commit
>>>
>>> 5986553 (TWILL-63) Speed up application launch time
>>>
>>> Let me see if can nail down to a particular change.
>>>
>>> Sam
>>>
>>>
>>>> On Mar 25, 2017, at 13:34, Sam William <[email protected]> wrote:
>>>>
>>>> HI Terence,
>>>> Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages for
>>>> my fat jars.
>>>>
>>>> SAm
>>>>> On Mar 25, 2017, at 12:31, Terence Yim <[email protected]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Haven't seen this error before. What is the version of Hadoop that the
>>>>> cluster is running with? Also, seems like the $HADOOP_CONF is not in the
>>>>> classpath as the FileContext is trying to use local file system instead
>>>>> of the distributed one.
>>>>>
>>>>> Terence
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On Mar 25, 2017, at 12:25 PM, Sam William <[email protected]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I have been using Twill for sometime now and I just tried to upgrade our
>>>>>> application from Twill-0.8.0 to 0.10.0. I havent made any kind of code
>>>>>> changes besides changing the Twill version string in the build script.
>>>>>> The application fails immediately and I see this on the RM UI. Any idea
>>>>>> why this could be happening?
>>>>>>
>>>>>> Diagnostics:
>>>>>> Application application_1484158548936_11154 failed 2 times due to AM
>>>>>> Container for appattempt_1484158548936_11154_000002 exited with
>>>>>> exitCode: -1000
>>>>>> For more detailed output, check application tracking page<> Then, click
>>>>>> on links to logs of each attempt.
>>>>>> Diagnostics: No such file or directory
>>>>>> ENOENT: No such file or directory
>>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
>>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
>>>>>> at
>>>>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:660)
>>>>>> at
>>>>>> org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:206)
>>>>>> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251)
>>>>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955)
>>>>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951)
>>>>>> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>>>>>> at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:951)
>>>>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419)
>>>>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:417)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>>>>>> at
>>>>>> org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:417)
>>>>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:363)
>>>>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Failing this attempt. Failing the application.
>>>>>>
>>>>>>
>>>>>> Sam
>>>>
>>>
>
