Terence, Yes the twill classes are included in the fat jar. I ll file a JIRA soon.
Thanks, Sam > On Mar 27, 2017, at 20:39, Terence Yim <cht...@gmail.com> wrote: > > Hi Sam, > > Do you have twill classes included in the fat jar as well? I think it might > be affecting the application jar that twill generate. Would you mind filing a > JIRA on this? > > Thanks, > Terence > >> On Mar 27, 2017, at 3:55 PM, Sam William <sampri...@gmail.com> wrote: >> >> I was able to make it work with this 1 line change. >> >> https://github.com/sampd/twill/compare/branch-0.10.0...sampd:v10_test?expand=1 >> >> <https://github.com/sampd/twill/compare/branch-0.10.0...sampd:v10_test?expand=1> >> >> We have a fat jar with all dependencies included for the application. Could >> that be the cause ? >> >> Sam >> >> >> >>> On Mar 27, 2017, at 15:49, Yuliya Feldman <yul...@dremio.com> wrote: >>> >>> Code of your application you want to be running in YARN I believe :) >>> >>> On Mon, Mar 27, 2017 at 3:28 PM, Sam William <sampri...@gmail.com> wrote: >>> >>>> Yes. 22 bytes looks like an empty zip file. Any idea what should there in >>>> the application jar file ? >>>> >>>> Sam >>>>> On Mar 27, 2017, at 13:22, Yuliya Feldman <yul...@dremio.com> wrote: >>>>> >>>>> File is very small - it may be nothing to do with file not found. Either >>>>> permissions or something else >>>>> >>>>> On Mon, Mar 27, 2017 at 1:17 PM, Sam William <sampri...@gmail.com> >>>> wrote: >>>>> >>>>>> I logged into the master host and looked at the nodemanager logs. It >>>> fails >>>>>> at localizing the application jar. The files are there in HDFS. I can >>>>>> even see it is able to copy the other files just fine (for example the >>>>>> launcher jar and runtime.config) >>>>>> >>>>>> -rw-r--r-- 3 sam supergroup 22 2017-03-27 12:47 >>>>>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- >>>>>> 44a506886fc1/Build-shards-GRE-bd5d893b401041edceec38c78f1ece >>>>>> c7-application.538b9590-d7f5-4121-824e-448a12a635c1.jar >>>>>> -rw-r--r-- 3 sam supergroup 5991970 2017-03-27 12:47 >>>>>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- >>>>>> 44a506886fc1/buil.b0458483-23ca-4243-89f6-d1a40210110d. >>>>>> -rw-r--r-- 3 sam supergroup 5725 2017-03-27 12:47 >>>>>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- >>>> 44a506886fc1/launcher. >>>>>> 4d7df397-5325-4a5f-8c95-ddcae99867f5.jar >>>>>> -rw-r--r-- 3 sam supergroup 1038 2017-03-27 12:47 >>>>>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- >>>>>> 44a506886fc1/localizeFiles.bbe5dc82-9fe9-4249-8964-df15212a1812.json >>>>>> -rw-r--r-- 3 sam supergroup 2072 2017-03-27 12:47 >>>>>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- >>>>>> 44a506886fc1/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar >>>>>> -rw-r--r-- 3 sam supergroup 48245414 2017-03-27 12:47 >>>>>> /user/sam/Build-shards-GRE/2f30b4ab-d9e1-48bd-9384- >>>>>> 44a506886fc1/twill.c765e4d8-958e-4811-b138-c4ef71e2a93e.jar >>>>>> >>>>>> >>>>>> 2017-03-27 12:47:45,632 INFO org.apache.hadoop.yarn.server. >>>>>> nodemanager.containermanager.localizer.LocalizedResource: Resource >>>>>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab- >>>>>> d9e1-48bd-9384-44a506886fc1/runtime.config.9dd1b585-c601- >>>>>> 40b7-8831-25383013eb1e.jar(->/data/8/yarn/nm/usercache/sam/ >>>>>> appcache/application_1484158548936_11282/filecache/ >>>>>> 11/runtime.config.9dd1b585-c601-40b7-8831-25383013eb1e.jar) >>>> transitioned >>>>>> from DOWNLOADING to LOCALIZED >>>>>> 2017-03-27 12:47:45,645 INFO org.apache.hadoop.yarn.server. >>>>>> nodemanager.containermanager.localizer.LocalizedResource: Resource >>>>>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab- >>>>>> d9e1-48bd-9384-44a506886fc1/launcher.4d7df397-5325-4a5f- >>>>>> 8c95-ddcae99867f5.jar(->/data/10/yarn/nm/usercache/sam/ >>>>>> appcache/application_1484158548936_11282/filecache/ >>>>>> 12/launcher.4d7df397-5325-4a5f-8c95-ddcae99867f5.jar) transitioned from >>>>>> DOWNLOADING to LOCALIZED >>>>>> 2017-03-27 12:47:45,651 WARN org.apache.hadoop.security. >>>> UserGroupInformation: >>>>>> PriviledgedActionException as:sam (auth:SIMPLE) cause:ENOENT: No such >>>> file >>>>>> or directory >>>>>> 2017-03-27 12:47:45,655 WARN org.apache.hadoop.yarn.server. >>>>>> nodemanager.containermanager.localizer.ResourceLocalizationService: { >>>>>> hdfs://pv34-search-dev/user/sam/Build-shards-GRE/2f30b4ab- >>>>>> d9e1-48bd-9384-44a506886fc1/Build-shards-GRE- >>>>>> bd5d893b401041edceec38c78f1ecec7-application.538b9590-d7f5- >>>> 4121-824e-448a12a635c1.jar, >>>>>> 1490644063924, ARCHIVE, null } failed: No such file or directory >>>>>> ENOENT: No such file or directory >>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native >>>>>> Method) >>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO. >>>>>> java:230) >>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission( >>>>>> RawLocalFileSystem.java:660) >>>>>> at org.apache.hadoop.fs.DelegateToFileSystem.setPermission( >>>>>> DelegateToFileSystem.java:206) >>>>>> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java: >>>> 251) >>>>>> at org.apache.hadoop.fs.FileContext$10.next( >>>> FileContext.java:955) >>>>>> at org.apache.hadoop.fs.FileContext$10.next( >>>> FileContext.java:951) >>>>>> at org.apache.hadoop.fs.FSLinkResolver.resolve( >>>>>> FSLinkResolver.java:90) >>>>>> at org.apache.hadoop.fs.FileContext.setPermission( >>>>>> FileContext.java:951) >>>>>> >>>>>> >>>>>>> On Mar 27, 2017, at 12:45, Sam William <sampri...@gmail.com> wrote: >>>>>>> >>>>>>> Hi Terence, >>>>>>> Im not able to get logs for these jobs. “yarn logs” command does nt >>>>>> return anything. >>>>>>> Sam >>>>>>>> On Mar 26, 2017, at 17:32, Terence Yim <cht...@gmail.com> wrote: >>>>>>>> >>>>>>>> Hi Sam, >>>>>>>> >>>>>>>> I guess it might be related to the missing of the Hadoop conf >>>> directory >>>>>> in the container classpath, such that the locationfactory constructed >>>> from >>>>>> the container side is not correct. Do you have access to the containers >>>>>> stdout file? It shows the classpath twill uses. >>>>>>>> >>>>>>>> Terence >>>>>>>> >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>>> On Mar 26, 2017, at 3:16 PM, Sam William <sampri...@gmail.com> >>>> wrote: >>>>>>>>> >>>>>>>>> It works with Twill-0.9.0. So far I have been able to narrow it down >>>>>> to one commit >>>>>>>>> >>>>>>>>> 5986553 (TWILL-63) Speed up application launch time >>>>>>>>> >>>>>>>>> Let me see if can nail down to a particular change. >>>>>>>>> >>>>>>>>> Sam >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Mar 25, 2017, at 13:34, Sam William <sampri...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> HI Terence, >>>>>>>>>> Our cloudera installation is CDH-5.7 and I use hadoop 2.3.0 packages >>>>>> for my fat jars. >>>>>>>>>> >>>>>>>>>> SAm >>>>>>>>>>> On Mar 25, 2017, at 12:31, Terence Yim <cht...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Haven't seen this error before. What is the version of Hadoop that >>>>>> the cluster is running with? Also, seems like the $HADOOP_CONF is not in >>>>>> the classpath as the FileContext is trying to use local file system >>>> instead >>>>>> of the distributed one. >>>>>>>>>>> >>>>>>>>>>> Terence >>>>>>>>>>> >>>>>>>>>>> Sent from my iPhone >>>>>>>>>>> >>>>>>>>>>>> On Mar 25, 2017, at 12:25 PM, Sam William <sampri...@gmail.com> >>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> I have been using Twill for sometime now and I just tried to >>>>>> upgrade our application from Twill-0.8.0 to 0.10.0. I havent made any >>>> kind >>>>>> of code changes besides changing the Twill version string in the build >>>>>> script. The application fails immediately and I see this on the RM UI. >>>> Any >>>>>> idea why this could be happening? >>>>>>>>>>>> >>>>>>>>>>>> Diagnostics: >>>>>>>>>>>> Application application_1484158548936_11154 failed 2 times due to >>>>>> AM Container for appattempt_1484158548936_11154_000002 exited with >>>>>> exitCode: -1000 >>>>>>>>>>>> For more detailed output, check application tracking page<> Then, >>>>>> click on links to logs of each attempt. >>>>>>>>>>>> Diagnostics: No such file or directory >>>>>>>>>>>> ENOENT: No such file or directory >>>>>>>>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native >>>>>> Method) >>>>>>>>>>>> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO. >>>>>> java:230) >>>>>>>>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission( >>>>>> RawLocalFileSystem.java:660) >>>>>>>>>>>> at org.apache.hadoop.fs.DelegateToFileSystem.setPermission( >>>>>> DelegateToFileSystem.java:206) >>>>>>>>>>>> at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:251) >>>>>>>>>>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:955) >>>>>>>>>>>> at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:951) >>>>>>>>>>>> at org.apache.hadoop.fs.FSLinkResolver.resolve( >>>>>> FSLinkResolver.java:90) >>>>>>>>>>>> at org.apache.hadoop.fs.FileContext.setPermission( >>>>>> FileContext.java:951) >>>>>>>>>>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload. >>>>>> java:419) >>>>>>>>>>>> at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload. >>>>>> java:417) >>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs( >>>>>> UserGroupInformation.java:1693) >>>>>>>>>>>> at org.apache.hadoop.yarn.util.FSDownload.changePermissions( >>>>>> FSDownload.java:417) >>>>>>>>>>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload. >>>> java:363) >>>>>>>>>>>> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload. >>>> java:60) >>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter. >>>>>> call(Executors.java:511) >>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>> ThreadPoolExecutor.java:1142) >>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>> ThreadPoolExecutor.java:617) >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>> Failing this attempt. Failing the application. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Sam >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>>> >>>> >>>> >> >
