Code of your application you want to be running in YARN I believe :) On Mon, Mar 27, 2017 at 3:28 PM, Sam William <[email protected]> wrote:
> Yes. 22 bytes looks like an empty zip file. Any idea what should there in > the application jar file ? > > Sam > > On Mar 27, 2017, at 13:22, Yuliya Feldman <[email protected]> wrote: > > > > File is very small - it may be nothing to do with file not found. Either > > permissions or something else > > > > On Mon, Mar 27, 2017 at 1:17 PM, Sam William <[email protected]> > wrote: > > > >> I logged into the master host and looked at the nodemanager logs. It > fails > >> at localizing the application jar. The files are there in HDFS. I can > >> even see it is able to copy the other files just fine (for example the > >> launcher jar and runtime.config) > >> > >> -rw-r--r-- 3 sam supergroup 22 2017-03-27 12:47 > >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- > >> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece > >> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar > >> -rw-r--r-- 3 sam supergroup 5991970 2017-03-27 12:47 > >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- > >> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d. > >> -rw-r--r-- 3 sam supergroup 5725 2017-03-27 12:47 > >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- > 44a506886fc1/launcher. > >> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar > >> -rw-r--r-- 3 sam supergroup 1038 2017-03-27 12:47 > >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- > >> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json > >> -rw-r--r-- 3 sam supergroup 2072 2017-03-27 12:47 > >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- > >> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar > >> -rw-r--r-- 3 sam supergroup 48245414 2017-03-27 12:47 > >> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- > >> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar > >> > >> > >> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server. > >> nodemanager.containermanager.localizer.LocalizedResource: Resource > >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab- > >> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601- > >> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/ > >> appcache/application_1484158548936_11282/filecache/ > >> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar) > transitioned > >> from DOWNLOADING to LOCALIZED > >> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server. > >> nodemanager.containermanager.localizer.LocalizedResource: Resource > >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab- > >> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f- > >> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/ > >> appcache/application_1484158548936_11282/filecache/ > >> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from > >> DOWNLOADING to LOCALIZED > >> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security. > UserGroupInformation: > >> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such > file > >> or directory > >> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server. > >> nodemanager.containermanager.localizer.ResourceLocalizationService: { > >> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab- > >> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE- > >> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5- > 4121-824e-448a12a635c1.jar, > >> 1490644063924, ARCHIVE, null } failed: No such file or directory > >> ENOENT: No such file or directory > >> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native > >> Method) > >> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO. > >> java:230) > >> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission( > >> RawLocalFileSystem.java:660) > >> at org.apache.hadoop.fs.DelegateToFileSystem.setPermission( > >> DelegateToFileSystem.java:206) > >> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java: > 251) > >> at org.apache.hadoop.fs.FileContext$10.next( > FileContext.java:955) > >> at org.apache.hadoop.fs.FileContext$10.next( > FileContext.java:951) > >> at org.apache.hadoop.fs.FSLinkResolver.resolve( > >> FSLinkResolver.java:90) > >> at org.apache.hadoop.fs.FileContext.setPermission( > >> FileContext.java:951) > >> > >> > >>> On Mar 27, 2017, at 12:45, Sam William <[email protected]> wrote: > >>> > >>> Hi Terence, > >>> Im not able to get logs for these jobs. “yarn logs” command does nt > >> return anything. > >>> Sam > >>>> On Mar 26, 2017, at 17:32, Terence Yim <[email protected]> wrote: > >>>> > >>>> Hi Sam, > >>>> > >>>> I guess it might be related to the missing of the Hadoop conf > directory > >> in the container classpath, such that the locationfactory constructed > from > >> the container side is not correct. Do you have access to the containers > >> stdout file? It shows the classpath twill uses. > >>>> > >>>> Terence > >>>> > >>>> Sent from my iPhone > >>>> > >>>>> On Mar 26, 2017, at 3:16 PM, Sam William <[email protected]> > wrote: > >>>>> > >>>>> It works with Twill-0.9.0. So far I have been able to narrow it down > >> to one commit > >>>>> > >>>>> 5986553 (TWILL-63) Speed up application launch time > >>>>> > >>>>> Let me see if can nail down to a particular change. > >>>>> > >>>>> Sam > >>>>> > >>>>> > >>>>>> On Mar 25, 2017, at 13:34, Sam William <[email protected]> wrote: > >>>>>> > >>>>>> HI Terence, > >>>>>> Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages > >> for my fat jars. > >>>>>> > >>>>>> SAm > >>>>>>> On Mar 25, 2017, at 12:31, Terence Yim <[email protected]> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> Haven't seen this error before. What is the version of Hadoop that > >> the cluster is running with? Also, seems like the $HADOOP_CONF is not in > >> the classpath as the FileContext is trying to use local file system > instead > >> of the distributed one. > >>>>>>> > >>>>>>> Terence > >>>>>>> > >>>>>>> Sent from my iPhone > >>>>>>> > >>>>>>>> On Mar 25, 2017, at 12:25 PM, Sam William <[email protected]> > >> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> I have been using Twill for sometime now and I just tried to > >> upgrade our application from Twill-0.8.0 to 0.10.0. I havent made any > kind > >> of code changes besides changing the Twill version string in the build > >> script. The application fails immediately and I see this on the RM UI. > Any > >> idea why this could be happening? > >>>>>>>> > >>>>>>>> Diagnostics: > >>>>>>>> Application application_1484158548936_11154 failed 2 times due to > >> AM Container for appattempt_1484158548936_11154_000002 exited with > >> exitCode: -1000 > >>>>>>>> For more detailed output, check application tracking page<> Then, > >> click on links to logs of each attempt. > >>>>>>>> Diagnostics: No such file or directory > >>>>>>>> ENOENT: No such file or directory > >>>>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native > >> Method) > >>>>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO. > >> java:230) > >>>>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission( > >> RawLocalFileSystem.java:660) > >>>>>>>> at org.apache.hadoop.fs.DelegateToFileSystem.setPermission( > >> DelegateToFileSystem.java:206) > >>>>>>>> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251) > >>>>>>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955) > >>>>>>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951) > >>>>>>>> at org.apache.hadoop.fs.FSLinkResolver.resolve( > >> FSLinkResolver.java:90) > >>>>>>>> at org.apache.hadoop.fs.FileContext.setPermission( > >> FileContext.java:951) > >>>>>>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload. > >> java:419) > >>>>>>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload. > >> java:417) > >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) > >>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) > >>>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs( > >> UserGroupInformation.java:1693) > >>>>>>>> at org.apache.hadoop.yarn.util.FSDownload.changePermissions( > >> FSDownload.java:417) > >>>>>>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload. > java:363) > >>>>>>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload. > java:60) > >>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>>>>>>> at java.util.concurrent.Executors$RunnableAdapter. > >> call(Executors.java:511) > >>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1142) > >>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:617) > >>>>>>>> at java.lang.Thread.run(Thread.java:745) > >>>>>>>> Failing this attempt. Failing the application. > >>>>>>>> > >>>>>>>> > >>>>>>>> Sam > >>>>>> > >>>>> > >>> > >> > >> > >
