[
https://issues.apache.org/jira/browse/MAPREDUCE-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193377#comment-13193377
]
Roman Shaposhnik commented on MAPREDUCE-3728:
---------------------------------------------
.bq May be I am missing something, but I don't understand this. The output-dir
is being created with null FilePermissions which becomes 0777.
I suspect this is not about *what* permissions it is create with, but the sheer
fact that permissions are set at all. Here's an example of how it goes wrong
when permissions are set explicitly. When that happens sticky bit gets lost.
Here's a trivial example. As you can see, when a subdir is created it is, at
first, retains the sticky bit. But when
I set permissions explicitly it gets cleared (as it should):
{noformat}
rvs@ahmed-laptop:/tmp$ cd /tmp
rvs@ahmed-laptop:/tmp$ mkdir sticky.dir
rvs@ahmed-laptop:/tmp$ sudo chgrp root sticky.dir
rvs@ahmed-laptop:/tmp$ sudo chmod g+s sticky.dir
rvs@ahmed-laptop:/tmp$ mkdir sticky.dir/subdir
rvs@ahmed-laptop:/tmp$ ls -l sticky.dir
total 4
drwxr-sr-x 2 rvs root 4096 2012-01-25 13:50 subdir
rvs@ahmed-laptop:/tmp$ chmod 777 sticky.dir/subdir
rvs@ahmed-laptop:/tmp$ ls -l sticky.dir
total 4
drwxrwxrwx 2 rvs root 4096 2012-01-25 13:50 subdir
{noformat}
.bq Instead I suspect that the umask for your testuser is strict. That my be
the reason why the files are getting created with 640. Can you please check?
The problem is not file permissions per se, but the fact that subdirs (starting
from output) have lost a group sticky bit and thus everything that is created
underneath them has user group as a default (and of course no longer retains
the sticky group bit on the subdirectories). But anyway, I've checked and the
umask for testuser is 0022
.bq Also, just to be sure, you are using DefaultContainerExecutor or
LinuxContainerExecutor? It is mostly the later, but just confirming.
This is LinuxContainerExecutor since, to the best of my knowledge, one has to
configure it for the fully secure cluster.
> ShuffleHandler can't access results when configured in a secure mode
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3728
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> While running the simplest of jobs (Pi) on MR2 in a fully secure
> configuration I have noticed that the job was failing on the reduce side with
> the following messages littering the nodemanager logs:
> {noformat}
> 2012-01-19 08:35:32,544 ERROR org.apache.hadoop.mapred.ShuffleHandler:
> Shuffle error
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> usercache/rvs/appcache/application_1326928483038_0001/output/attempt_1326928483038_0001_m_000003_0/file.out.index
> in any of the configured local directories
> {noformat}
> While digging further I found out that the permissions on the files/dirs were
> prohibiting nodemanager (running under the user yarn) to access these files:
> {noformat}
> $ ls -l
> /data/3/yarn/usercache/testuser/appcache/application_1327102703969_0001/output/attempt_1327102703969_0001_m_000001_0
> -rw-r----- 1 testuser testuser 28 Jan 20 15:41 file.out
> -rw-r----- 1 testuser testuser 32 Jan 20 15:41 file.out.index
> {noformat}
> Digging even further revealed that the group-sticky bit that was faithfully
> put on all the subdirectories between testuser and
> application_1327102703969_0001 was gone from output and
> attempt_1327102703969_0001_m_000001_0.
> Looking into how these subdirectories are created
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs())
> {noformat}
> // $x/usercache/$user/appcache/$appId/filecache
> Path appFileCacheDir = new Path(appBase, FILECACHE);
> appsFileCacheDirs[i] = appFileCacheDir.toString();
> lfs.mkdir(appFileCacheDir, null, false);
> // $x/usercache/$user/appcache/$appId/output
> lfs.mkdir(new Path(appBase, OUTPUTDIR), null, false);
> {noformat}
> Reveals that lfs.mkdir ends up manipulating permissions and thus clears
> sticky bit from output and filecache.
> At this point I'm at a loss about how this is supposed to work. My
> understanding was
> that the whole sequence of events here was predicated on a sticky bit set so
> that daemons running under the user yarn (default group yarn) can have access
> to the resulting files and subdirectories down at output and below. Please let
> me know if I'm missing something or whether this is just a bug that needs to
> be fixed.
> On a related note, when the shuffle side of the Pi job failed the job itself
> didn't.
> It went into the endless loop and only exited when it exhausted all the local
> storage
> for the log files (at which point the nodemanager died and thus the job
> ended). Perhaps
> this is even more serious side effect of this issue that needs to be
> investigated
> separately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
