[
https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793937#action_12793937
]
Hemanth Yamijala commented on MAPREDUCE-896:
--------------------------------------------
Some more details on the test failure in TestServiceLevelAuthorization:
The problem is with TestMiniMRWithDFS.checkTaskDirectories. As explained above,
this method verifies an exact list of localized directories on a tasktracker.
To do so, it seems to wait until the TTs become idle. But idleness is only a
function of the number of active tasks. When tasks complete, the TT adds
directories to be deleted to the CleanupQueue, which by default, asynchronously
deletes them. Clearly that means that waiting for the TTs to become idle is not
a sufficient pre-condition for the testing of localized directory contents.
Like I explained, this problem exists on trunk as well. But this patch can
increase the chances of hitting it more often, because it does more work in the
asynchronous portion. Given that, I am not comfortable committing it,
particularly as there's no blazing hurry for this patch right now. We do run a
risk of the patch going stale, but I am relying on the holiday season to slow
people down (*smile*). More importantly, we'll not make the test scene worse
than it already is.
As regards the fix, a simple solution may be to configure the MiniMRCluster
with an inline cleanup queue whenever we want to check the deletion of
localized files and directories - essentially in all test cases that use
TestMiniMRWithDFS.checkTaskDirectories. Fortunately, there are only four of
them right now. Another option could be to wait until the cleanup queue is
empty. The disadvantage with this approach is that it uses the much maligned
"sleep(100) until some condition is satisfied" pattern. I would use the former
approach instead, which has worked well elsewhere. It may slow the tests down a
tad bit, but hopefully not too badly.
> Users can set non-writable permissions on temporary files for TT and can
> abuse disk usage.
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-896
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-896
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.21.0
> Reporter: Vinod K V
> Assignee: Ravi Gummadi
> Fix For: 0.21.0
>
> Attachments: MR-896.patch, MR-896.v1.patch, MR-896.v2.patch,
> MR-896.v3.patch, MR-896.v4.patch, MR-896.v5.patch, MR-896.v6.patch,
> y896.v1.patch, y896.v2.1.fix.patch, y896.v2.1.fix.v1.patch,
> y896.v2.1.fix.v2.patch, y896.v2.1.patch, y896.v2.patch
>
>
> As of now, irrespective of the TaskController in use, TT itself does a full
> delete on local files created by itself or job tasks. This step, depending
> upon TT's umask and the permissions set by files by the user, for e.g in
> job-work/task-work or child.tmp directories, may or may not go through
> successful completion fully. Thus is left an opportunity for abusing disk
> space usage either accidentally or intentionally by TT/users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
