yes i checked the code ,and find the Exception from lfs.mkdir(userFileCacheDir, null, false);
also find the AM located in CHBM224 ,all will failed but,AM located in CHBM223,all success in CHBM224 # ls -l /data/mrlocal/1/yarn/ total 8 drwxrwxrwx 5 yarn yarn 4096 Nov 5 20:50 local drwxr-xr-x 3 yarn yarn 4096 Jan 3 15:57 logs # ls -l /data/mrlocal/2/yarn/ total 8 drwxrwxrwx 5 yarn yarn 4096 Nov 5 20:50 local drwxr-xr-x 3 yarn yarn 4096 Jan 3 15:57 logs in CHBM223 # ls /data/mrlocal/1/yarn/ -l total 8 drwxr-xr-x 5 yarn yarn 4096 Nov 5 20:51 local drwxr-xr-x 3 yarn yarn 4096 Jan 3 15:46 logs # ls /data/mrlocal/2/yarn/ -l total 8 drwxr-xr-x 5 yarn yarn 4096 Nov 5 20:51 local drwxr-xr-x 3 yarn yarn 4096 Jan 3 15:46 logs i also find if i let abnormal node (CHBM224) run and shutdown the other normal node ,when i submit a MR job use hive ,and the dir "/data/mrlocal/2/yarn/local/usercache/hive/filecache" ,it's mode will flush to 710 ,even i change the file to 755,but i test on a normal node (open one normal node ,and shutdown others) ,the dir mode will not changed # ls -l /data/mrlocal/2/yarn/local/usercache/hive/ total 16 drwx--x--- 7 yarn yarn 4096 Jan 3 16:30 appcache drwx--x--- 148 yarn yarn 12288 Jan 3 10:03 filecache On Fri, Jan 3, 2014 at 3:52 PM, Bing Jiang <jiangbinglo...@gmail.com> wrote: > Could you check your yarn-local directory authority? From the diagnosis, > error occurs at mkdir in local directory. > I guess something wrong with local direcotry which is set as yarn local > dir. > > > > 2014/1/3 ch huang <justlo...@gmail.com> > >> hi, i submit a MR job through hive ,but when it run stage-2 ,it failed >> why? >> >> it seems permission problem ,but i do not know which dir cause the >> problem >> >> Application application_1388730279827_0035 failed 1 times due to AM >> Container for appattempt_1388730279827_0035_000001 exited with exitCode: >> -1000 due to: EPERM: Operation not permitted at >> org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) at >> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581) >> at >> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:388) >> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1041) at >> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150) >> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:190) at >> org.apache.hadoop.fs.FileContext$4.next(FileContext.java:698) at >> org.apache.hadoop.fs.FileContext$4.next(FileContext.java:695) at >> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325) >> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:695) at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:385) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:861) >> .Failing this attempt.. Failing the application. >> > > > > -- > Bing Jiang > Tel:(86)134-2619-1361 > weibo: http://weibo.com/jiangbinglover > BLOG: www.binospace.com > BLOG: http://blog.sina.com.cn/jiangbinglover > Focus on distributed computing, HDFS/HBase >