[jira] Updated: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[
https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated MAPREDUCE-1422:
--
Attachment: mapreduce-1422-test-v1.0.patch
Attaching an example test-job that can be run on a cluster to verify the fix.
Here are the steps to verify :
1) _Reproduce the bug_
{code}
1) Apply this patch.
2) Deploy the cluster
3) run './bin/hadoop jar build/hadoop-mapred-test-*.jar badjobexample -m 1 -mt
1 -recordt 1'
4) While the job is running, check ${job-work-dir}/subDir/ on the tasktracker
5) Even after the job completes, some files are left behind in the job-work-dir.
{code}
2) _Verify the fix_
Apply the
[patch|https://issues.apache.org/jira/secure/attachment/12436775/mapreduce-1422-v1.4.2.patch]
along with the attached patch and repeat the steps mentioned above. With this
patch, the job-work-dir is properly cleanup as expected.
Note : job-work-dir for the default case would look something like
{mapred.local.dir}/taskTracker/{user.name}/jobcache/{job.id}/work/
> Changing permissions of files/dirs under job-work-dir may be needed sothat
> cleaning up of job-dir in all mapred-local-directories succeeds always
> -
>
> Key: MAPREDUCE-1422
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Reporter: Ravi Gummadi
>Assignee: Amar Kamat
> Attachments: mapreduce-1422-test-v1.0.patch,
> mapreduce-1422-v1.4.2.patch
>
>
> After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller
> binary is launched for changing permissions of taskAttemptDir and taskWorkDir
> before cleaning up of these directories sothat cleanup will be succeeded even
> if user had created files/dirs under taskAttemptDir or taskWorkDir with
> non-writable permissions. Users can't create files/dirs under job-dir
> directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions
> and user can create files/dirs under job-work-dir with non-writable
> permissions, Changing permissions of files/dirs under job-work-dir may be
> needed sothat cleaning up of job-dir in all mapred-local-directories succeeds
> always.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1519) RaidNode fails to create new parity file if an older version already exists
[ https://issues.apache.org/jira/browse/MAPREDUCE-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840042#action_12840042 ] Hudson commented on MAPREDUCE-1519: --- Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/]) > RaidNode fails to create new parity file if an older version already exists > --- > > Key: MAPREDUCE-1519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1519 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1519.patch > > > When RaidNode tries to recreate a parity file for a source file that has been > modified (recreated) recently, it crashes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1537) TestDelegationTokenRenewal fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840040#action_12840040 ] Hudson commented on MAPREDUCE-1537: --- Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/]) > TestDelegationTokenRenewal fails > > > Key: MAPREDUCE-1537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1537 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: MR-1537.1.patch, MR-1537.2.patch, MR-1537.3.patch > > > TestDelegationTokenRenewal does not compile in trunk. > The reason is that DelegationTokenSecretManager in hdfs requires namesystem > in constructor. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1503) Push HADOOP-6551 into MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840037#action_12840037 ] Hudson commented on MAPREDUCE-1503: --- Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/]) > Push HADOOP-6551 into MapReduce > --- > > Key: MAPREDUCE-1503 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1503 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.22.0 > > Attachments: m-1503.patch > > > We need to throw readable exceptions instead of returning false. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1430) JobTracker should be able to renew delegation tokens for the jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840036#action_12840036 ] Hudson commented on MAPREDUCE-1430: --- Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/]) > JobTracker should be able to renew delegation tokens for the jobs > - > > Key: MAPREDUCE-1430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1430 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: 1430-BP20-Add.patch, 1430-dd1.patch, 1430-dd2.patch, > 1430-dd3.patch, 1430-dd4-BP20.patch, 1430-dd4.patch, MAPREDUCE-1430-12.patch, > MAPREDUCE-1430-13.patch, MAPREDUCE-1430-14.patch, MAPREDUCE-1430-5.patch, > MAPREDUCE-1430-6.patch, MAPREDUCE-1430-8.patch > > > JobTracker should automatically renew delegation tokens for the jobs it is > currently running. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1307) Introduce the concept of Job Permissions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840038#action_12840038 ] Hudson commented on MAPREDUCE-1307: --- Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/]) > Introduce the concept of Job Permissions > > > Key: MAPREDUCE-1307 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1307 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security >Reporter: Devaraj Das >Assignee: Vinod K V > Fix For: 0.22.0 > > Attachments: 1307-early-1.patch, MAPREDUCE-1307-20100210.txt, > MAPREDUCE-1307-20100211.txt, MAPREDUCE-1307-20100215.txt, > MAPREDUCE-1307-20100217.txt, MAPREDUCE-1307-20100226.1-ydist.txt, > MAPREDUCE-1307-20100227-ydist.txt > > > It would be good to define the notion of job permissions analogous to file > permissions. Then the JobTracker can restrict who can "read" (e.g. look at > the job page) or "modify" (e.g. kill) jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1065) Modify the mapred tutorial documentation to use new mapreduce api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840035#action_12840035 ] Aaron Kimball commented on MAPREDUCE-1065: -- The test failure, of course, is unrelated > Modify the mapred tutorial documentation to use new mapreduce api. > -- > > Key: MAPREDUCE-1065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1065 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Aaron Kimball >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1065.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[
https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840032#action_12840032
]
Ravi Gummadi commented on MAPREDUCE-927:
>Creation of the localFS should not be in a static block. In the past also, we
>did this and realized it creates a kind of circular initialization of loggers
>and results in NPE while creating log objects which can be seen in task-logs.
>>Can you elaborate this? How will static initialization code result into
>>circular initialization of loggers? How will this result into NPE?
LogFactory.getLog() in Configuration.java goes through log4j.Logger.getLogger()
and calls activateOptions() that again calls TaskLog.getTaskLogFile(). Since
TaskLog.getTaskLogFile() is a static method, the static block in TaskLog.java
gets executed and that calls new Configuration(). Thus, from static LOG =
LogFactory.getLog(Configuration.class) to we came back to constructor of
Configuration() and now the constructor is getting executed(even though the
static blocks of Configuration.java are not finished execution). The
constructor of Configuration() was doing LOG.isDebugEnabled(){.} and LOG is
null now and thus resulting in NPE.
Hope that helps.
> Cleanup of task-logs should happen in TaskTracker instead of the Child
> --
>
> Key: MAPREDUCE-927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-927
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: security, tasktracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: Amareshwari Sriramadasu
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: patch-927-1.txt, patch-927.txt
>
>
> Task logs' cleanup is being done in Child now. This is undesirable atleast
> for two reasons: 1) failures while cleaning up will affect the user's tasks,
> and 2) the task's wall time will get affected due to operations that TT
> actually should own.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1547) Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840026#action_12840026 ] Giridharan Kesavan commented on MAPREDUCE-1547: --- h3.grid.sp2.yahoo.net machine is re-jumped with new ubuntu image. It's missing some build related packages, Im working on installing them and trying offline builds.. -Giri > Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails > -- > > Key: MAPREDUCE-1547 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1547 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan >Assignee: Giridharan Kesavan > > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/243/console > Build Hadoop-Mapreduce-trunk fails with this message > BUILD FAILED > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:1382: > Execute failed: java.io.IOException: Cannot run program "autoreconf" (in > directory > "/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/c++/utils"): > java.io.IOException: error=2, No such file or directory > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/251/console > Build Mapreduce-trunk-Commit fails with this message: > /bin/bash /tmp/hudson2120484997034746272.sh > bash: > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/nightly/commitBuild.sh: > No such file or directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1547) Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan reassigned MAPREDUCE-1547: - Assignee: Giridharan Kesavan > Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails > -- > > Key: MAPREDUCE-1547 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1547 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan >Assignee: Giridharan Kesavan > > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/243/console > Build Hadoop-Mapreduce-trunk fails with this message > BUILD FAILED > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:1382: > Execute failed: java.io.IOException: Cannot run program "autoreconf" (in > directory > "/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/c++/utils"): > java.io.IOException: error=2, No such file or directory > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/251/console > Build Mapreduce-trunk-Commit fails with this message: > /bin/bash /tmp/hudson2120484997034746272.sh > bash: > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/nightly/commitBuild.sh: > No such file or directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1547) Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iyappan Srinivasan updated MAPREDUCE-1547: -- Description: http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/243/console Build Hadoop-Mapreduce-trunk fails with this message BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:1382: Execute failed: java.io.IOException: Cannot run program "autoreconf" (in directory "/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/c++/utils"): java.io.IOException: error=2, No such file or directory http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/251/console Build Mapreduce-trunk-Commit fails with this message: /bin/bash /tmp/hudson2120484997034746272.sh bash: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/nightly/commitBuild.sh: No such file or directory was: http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/243/console Build fails with thsi message BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:1382: Execute failed: java.io.IOException: Cannot run program "autoreconf" (in directory "/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/c++/utils"): java.io.IOException: error=2, No such file or directory Summary: Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails (was: Build Hadoop-Mapreduce-trunk fails) > Build Hadoop-Mapreduce-trunk and Mapreduce-trunk-Commit fails > -- > > Key: MAPREDUCE-1547 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1547 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/243/console > Build Hadoop-Mapreduce-trunk fails with this message > BUILD FAILED > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:1382: > Execute failed: java.io.IOException: Cannot run program "autoreconf" (in > directory > "/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/c++/utils"): > java.io.IOException: error=2, No such file or directory > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/251/console > Build Mapreduce-trunk-Commit fails with this message: > /bin/bash /tmp/hudson2120484997034746272.sh > bash: > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/nightly/commitBuild.sh: > No such file or directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1547) Build Hadoop-Mapreduce-trunk fails
Build Hadoop-Mapreduce-trunk fails -- Key: MAPREDUCE-1547 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1547 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/243/console Build fails with thsi message BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:1382: Execute failed: java.io.IOException: Cannot run program "autoreconf" (in directory "/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/c++/utils"): java.io.IOException: error=2, No such file or directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1065) Modify the mapred tutorial documentation to use new mapreduce api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840015#action_12840015 ] Hadoop QA commented on MAPREDUCE-1065: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437089/MAPREDUCE-1065.patch against trunk revision 916823. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/492/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/492/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/492/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/492/console This message is automatically generated. > Modify the mapred tutorial documentation to use new mapreduce api. > -- > > Key: MAPREDUCE-1065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1065 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Aaron Kimball >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1065.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840008#action_12840008 ] Amareshwari Sriramadasu commented on MAPREDUCE-927: --- bq. The logscleanup thread is not joined/killed in the TaskTracker.close(). So, there will be zombie threads in the system on a re-init and may well interfere with the new thread. I just realized that the thread is initialized in intialize() method. It should be moved to constructor. > Cleanup of task-logs should happen in TaskTracker instead of the Child > -- > > Key: MAPREDUCE-927 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-927 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security, tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu >Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-927-1.txt, patch-927.txt > > > Task logs' cleanup is being done in Child now. This is undesirable atleast > for two reasons: 1) failures while cleaning up will affect the user's tasks, > and 2) the task's wall time will get affected due to operations that TT > actually should own. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840003#action_12840003 ] Amareshwari Sriramadasu commented on MAPREDUCE-927: --- bq. Creation of the localFS should not be in a static block. In the past also, we did this and realized it creates a kind of circular initialization of loggers and results in NPE while creating log objects which can be seen in task-logs. Can you elaborate this? How will static initialization code result into circular initialization of loggers? How will this result into NPE? bq. Also can't we simply construct a local-filesystem here itself, instead of calling TaskLog.getLocalFileSystem()? I'm getting the log file system through TaskLog.getLogFileSystem() to make sure that it is the same FileSystem used in TaskLog creation and deletion. bq. Rename the class to TaskLogsMonitor, so that we are consistent going forward with MAPREDUCE-1100. bq. The logscleanup thread is not joined/killed in the TaskTracker.close(). So, there will be zombie threads in the system on a re-init and may well interfere with the new thread. TaskLogMonitor introduced in MAPREDUCE-1100 is not a daemon thread, but TaskLogCleanupThread needs to be. I don't think we can merge the functionality of TaskLogCleanupThread and TaskLogMonitor into one thread. TaskLogCleanupThread is not started after every re-init. It is started only once similar to taskCleanupThread, directoryCleanupThread. So, this should not be joined in TaskTracker.close(). > Cleanup of task-logs should happen in TaskTracker instead of the Child > -- > > Key: MAPREDUCE-927 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-927 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security, tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu >Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-927-1.txt, patch-927.txt > > > Task logs' cleanup is being done in Child now. This is undesirable atleast > for two reasons: 1) failures while cleaning up will affect the user's tasks, > and 2) the task's wall time will get affected due to operations that TT > actually should own. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1542) Deprecate mapred.permissions.supergroup in favor of hadoop.cluster.administrators
[
https://issues.apache.org/jira/browse/MAPREDUCE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1284#action_1284
]
Vinod K V commented on MAPREDUCE-1542:
--
Allen, configuration of commong components is bound to be like this because of
the fact that we have a common codebase/project which both MAPREDUCE and HDFS
use. The configuration of the common components like Http servers, RPC
servers/clients will not have mapred/hdfs specific naming conventions. For
e.g., I don't see {{ipc.server.listen.queue.size}} or
{{ipc.client.connect.max.retries}} being split into mapred and hdfs specific
configuration items despite the fact that JT and NN need different values. I
think {{hadoop.cluster.administrators}} also falls under the same category.
> Deprecate mapred.permissions.supergroup in favor of
> hadoop.cluster.administrators
> -
>
> Key: MAPREDUCE-1542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Reporter: Vinod K V
> Fix For: 0.22.0
>
>
> HADOOP-6568 added the configuration {{hadoop.cluster.administrators}} through
> which admins can configure who the superusers/supergroups for the cluster
> are. MAPREDUCE itself already has {{mapred.permissions.supergroup}} (which is
> just a single group). As agreed upon at HADOOP-6568, this should be
> deprecated in favor of {{hadoop.cluster.administrators}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1546) URL to JT webconsole for running task and task history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1546: -- Description: We should redirect taskdetails.jsp to taskdetailshistory.jsp if the task is gone just like MAPREDUCE-1185 (was: It will be good if there is a link goes to the history page when the job/task is gone from JobTracker memory. That way we don't have to look for the history link again. ) Summary: URL to JT webconsole for running task and task history should be the same (was: jobdetails.jsp and taskdetials.jsp should show links to the corresponding jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone) > URL to JT webconsole for running task and task history should be the same > - > > Key: MAPREDUCE-1546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1546 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > > We should redirect taskdetails.jsp to taskdetailshistory.jsp if the task is > gone just like MAPREDUCE-1185 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1546) jobdetails.jsp and taskdetials.jsp should show links to the corresponding jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone
[ https://issues.apache.org/jira/browse/MAPREDUCE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1283#action_1283 ] Scott Chen commented on MAPREDUCE-1546: --- That is awesome, Amareshwari. That's exactly what I need. I will change this one for taskdetails.jsp only. > jobdetails.jsp and taskdetials.jsp should show links to the corresponding > jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone > -- > > Key: MAPREDUCE-1546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1546 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > > It will be good if there is a link goes to the history page when the job/task > is gone from JobTracker memory. > That way we don't have to look for the history link again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839994#action_12839994
]
Amareshwari Sriramadasu commented on MAPREDUCE-1522:
+1 Changes look fine.
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
> m1522_20100301.patch, m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1546) jobdetails.jsp and taskdetials.jsp should show links to the corresponding jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone
[ https://issues.apache.org/jira/browse/MAPREDUCE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839992#action_12839992 ] Amareshwari Sriramadasu commented on MAPREDUCE-1546: jobdetails.jsp will be automatically re-directed to jobdetailshistory.jsp after MAPREDUCE-1185. > jobdetails.jsp and taskdetials.jsp should show links to the corresponding > jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone > -- > > Key: MAPREDUCE-1546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1546 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > > It will be good if there is a link goes to the history page when the job/task > is gone from JobTracker memory. > That way we don't have to look for the history link again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1510) RAID should regenerate parity files if they get deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839968#action_12839968 ] Rodrigo Schmidt commented on MAPREDUCE-1510: Passed all contrib unit tests. I also verified the logs and confirmed that the RaidNode was binding to "random" free ports, different from the default one. This patch should be fine to be committed, if it passes human review. > RAID should regenerate parity files if they get deleted > --- > > Key: MAPREDUCE-1510 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1510 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Attachments: MAPREDUCE-1510.1.patch, MAPREDUCE-1510.2.patch, > MAPREDUCE-1510.patch > > > Currently, if a source file has a replication factor lower or equal to that > expected by RAID, the file is skipped and no parity file is generated. I > don't think this is a good behavior since parity files can get wrongly > deleted, leaving the source file with a low replication factor. In that case, > raid should be able to recreate the parity file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1546) jobdetails.jsp and taskdetials.jsp should show links to the corresponding jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone
jobdetails.jsp and taskdetials.jsp should show links to the corresponding jobdetailshistory.jsp and taskdetailshistory.jsp if the task/job is gone -- Key: MAPREDUCE-1546 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1546 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Fix For: 0.22.0 It will be good if there is a link goes to the history page when the job/task is gone from JobTracker memory. That way we don't have to look for the history link again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1510) RAID should regenerate parity files if they get deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839952#action_12839952 ] Rodrigo Schmidt commented on MAPREDUCE-1510: Passed all unit tests except [junit] Test org.apache.hadoop.mapred.TestMiniMRLocalFS FAILED But this one is broken in trunk and I don't modify anything related to it, so it doesn't count. Now I'm running the contrib tests. > RAID should regenerate parity files if they get deleted > --- > > Key: MAPREDUCE-1510 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1510 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Attachments: MAPREDUCE-1510.1.patch, MAPREDUCE-1510.2.patch, > MAPREDUCE-1510.patch > > > Currently, if a source file has a replication factor lower or equal to that > expected by RAID, the file is skipped and no parity file is generated. I > don't think this is a good behavior since parity files can get wrongly > deleted, leaving the source file with a low replication factor. In that case, > raid should be able to recreate the parity file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839924#action_12839924 ] Hadoop QA commented on MAPREDUCE-1466: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437051/mr-1466-trunk-v1.patch against trunk revision 916823. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/491/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/491/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/491/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/491/console This message is automatically generated. > FileInputFormat should save #input-files in JobConf > --- > > Key: MAPREDUCE-1466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 0.22.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1466_yhadoop20-1.patch, > MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20-3.patch, > MAPREDUCE-1466_yhadoop20.patch, mr-1466-trunk-v1.patch > > > We already track the amount of data consumed by MR applications > (MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the > client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy > to stick in the JobConf during job-submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1510) RAID should regenerate parity files if they get deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839921#action_12839921 ] Rodrigo Schmidt commented on MAPREDUCE-1510: Hudson is taking too long to generate a report on this one, so I'm doing the testing myself. ant test-patch returned the following: [exec] There appear to be 0 release audit warnings before the patch and 0 release audit warnings after applying the patch. [exec] [exec] [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 12 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == Now I'm running the unit tests > RAID should regenerate parity files if they get deleted > --- > > Key: MAPREDUCE-1510 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1510 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Attachments: MAPREDUCE-1510.1.patch, MAPREDUCE-1510.2.patch, > MAPREDUCE-1510.patch > > > Currently, if a source file has a replication factor lower or equal to that > expected by RAID, the file is skipped and no parity file is generated. I > don't think this is a good behavior since parity files can get wrongly > deleted, leaving the source file with a low replication factor. In that case, > raid should be able to recreate the parity file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1155) Streaming tests swallow exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-1155: --- Attachment: TestStreamingExitStatus.patch Attaching a patch for TestStreamingExitStatus test only (factored out the relevant part from the patch that got committed). > Streaming tests swallow exceptions > -- > > Key: MAPREDUCE-1155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1155 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce-1155.patch, mapreduce-1155.patch, > mapreduce-1155.txt, mapreduce-1155.txt, TestStreamingExitStatus.patch > > > Many of the streaming tests (including TestMultipleArchiveFiles) catch > exceptions and print their stack trace rather than failing the job. This > means that tests do not fail even when the job fails. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
Add 'first-task-launched' to job-summary Key: MAPREDUCE-1545 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 It would be useful to track 'first-task-launched' time to job-summary for better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1544) Miscellaneous improvements to HTML markup for web UIs
Miscellaneous improvements to HTML markup for web UIs - Key: MAPREDUCE-1544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1544 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Todd Lipcon Priority: Minor The Web UIs have various bits of bad markup (eg missing sections, some pages missing CSS links, inconsistent td vs th for table headings). We should fix this up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1031) ant tar target doens't seem to compile tests in contrib projects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839840#action_12839840 ] Aaron Kimball commented on MAPREDUCE-1031: -- The failing test has been doing so for a few builds now; unrelated to this patch. > ant tar target doens't seem to compile tests in contrib projects > > > Key: MAPREDUCE-1031 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1031 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Aaron Kimball >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1031-branch21.patch, MAPREDUCE-1031.patch > > > ant tar shouldn't be skipping contrib tests. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1328) contrib/index - modify build / ivy files as appropriate
[ https://issues.apache.org/jira/browse/MAPREDUCE-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1328: - Fix Version/s: (was: 0.20.2) > contrib/index - modify build / ivy files as appropriate > - > > Key: MAPREDUCE-1328 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1328 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/index >Affects Versions: 0.20.2 >Reporter: Kay Kay > Attachments: MAPREDUCE-1328.patch > > > The build / ivy.xml files in its current state does not seem to launch > successfully due to missing dependencies. > Added dependency on : hadoop-core-test / hadoop-hdfs-test . > Also the junit classpath is set to include the files retrieved by ivy , > specific to the index project. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1428) Make block size and the size of archive created files configurable.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-1428: -- Attachment: BinaryFileGenerator.java Here is a program to generate files with different sizes. I used it to test archive with max_k=32. BinaryFileGenerator.java: generate files with sizes 0, 2^k-1, 2^k and 2^k+1 for k=1,.., max_k. > Make block size and the size of archive created files configurable. > --- > > Key: MAPREDUCE-1428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1428 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.22.0 > > Attachments: BinaryFileGenerator.java > > > Currently the block size used by archives is the default block size of the > hdfs filesystem. We need to make it configurable so that the block size can > be higher for the part files that archives create. > Also, we need to make the size of part files in archives configurable again > to make it bigger in size and create less number of such files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1031) ant tar target doens't seem to compile tests in contrib projects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839820#action_12839820 ] Hadoop QA commented on MAPREDUCE-1031: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436923/MAPREDUCE-1031.patch against trunk revision 916823. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2252 javac compiler warnings (more than the trunk's current 2219 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/490/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/490/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/490/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/490/console This message is automatically generated. > ant tar target doens't seem to compile tests in contrib projects > > > Key: MAPREDUCE-1031 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1031 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Aaron Kimball >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1031-branch21.patch, MAPREDUCE-1031.patch > > > ant tar shouldn't be skipping contrib tests. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1522:
--
Assignee: Tsz Wo (Nicholas), SZE
Status: Patch Available (was: Open)
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
> m1522_20100301.patch, m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1522:
--
Status: Open (was: Patch Available)
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
> Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
> m1522_20100301.patch, m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1522:
--
Attachment: (was: m1522_20100224.patch)
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
> Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
> m1522_20100301.patch, m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1522:
--
Attachment: m1522_20100301.patch
> m1522_20100224.patch: incorporated both review comments.
Oops, the file should be m1522_20100301.patch.
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
> Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
> m1522_20100301.patch, m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1522:
--
Attachment: m1522_20100224.patch
Amareshwari, thanks for the review.
m1522_20100224.patch: incorporated both review comments.
Without the changes in FileInputFormat, it will fail on the new test as
following.
{noformat}
defaultfs.getUri() = s3://abc:x...@hostname
original = file:/foo
results = [file://abc:x...@hostname/foo]
- ---
Testcase: testAddInputPath took 0.704 sec
FAILED
expected: but was:
junit.framework.AssertionFailedError: expected: but
was:
at
org.apache.hadoop.mapreduce.lib.input.TestFileInputFormat.testAddInputPath(TestFileInputFormat.java:54)
{noformat}
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
> Attachments: m1522_20100224.patch, m1522_20100224.patch,
> m1522_20100224_0.20.patch, m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Patch Available (was: Open) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1482) Better handling of task diagnostic information stored in the TaskInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-1482: -- Status: Patch Available (was: Open) > Better handling of task diagnostic information stored in the TaskInProgress > --- > > Key: MAPREDUCE-1482 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1482 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amar Kamat >Assignee: Amar Kamat > Attachments: mapreduce-1482-v1.1.patch, mapreduce-1482-v1.2.patch > > > Task diagnostic information can be very large at times eating up Jobtracker's > memory. There should be some way to avoid storing large error strings in > JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1542) Deprecate mapred.permissions.supergroup in favor of hadoop.cluster.administrators
[
https://issues.apache.org/jira/browse/MAPREDUCE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839784#action_12839784
]
Allen Wittenauer commented on MAPREDUCE-1542:
-
Every time someone says "oh you can use a separate config file" I need to start
thanking them. It just means that Hadoop is that much more operationally
complex and helps to make sure that I'll have future employment. :)
> Deprecate mapred.permissions.supergroup in favor of
> hadoop.cluster.administrators
> -
>
> Key: MAPREDUCE-1542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Reporter: Vinod K V
> Fix For: 0.22.0
>
>
> HADOOP-6568 added the configuration {{hadoop.cluster.administrators}} through
> which admins can configure who the superusers/supergroups for the cluster
> are. MAPREDUCE itself already has {{mapred.permissions.supergroup}} (which is
> just a single group). As agreed upon at HADOOP-6568, this should be
> deprecated in favor of {{hadoop.cluster.administrators}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1455) Authorization for servlets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839754#action_12839754 ] Ravi Gummadi commented on MAPREDUCE-1455: - Unit tests passed on my local machine(except the known failures of MAPREDUCE-1421 and MAPREDUCE-1520). ant test-patch gave: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 11 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. > Authorization for servlets > -- > > Key: MAPREDUCE-1455 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1455 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security, tasktracker >Reporter: Devaraj Das >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1455.20S.2.fix.patch, 1455.20S.2.patch, 1455.patch, > 1455.v1.patch, 1455.v2.patch, 1455.v3.patch, 1455.v4.1.patch, > 1455.v4.2.patch, 1455.v4.patch > > > This jira is about building the authorization for servlets (on top of > MAPREDUCE-1307). That is, the JobTracker/TaskTracker runs authorization > checks on web requests based on the configured job permissions. For e.g., if > the job permission is 600, then no one except the authenticated user can look > at the job details via the browser. The authenticated user in the servlet can > be obtained using the HttpServletRequest method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1430) JobTracker should be able to renew delegation tokens for the jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839734#action_12839734 ] Hadoop QA commented on MAPREDUCE-1430: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436453/1430-dd1.patch against trunk revision 912196. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/467/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/467/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/467/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/467/console This message is automatically generated. > JobTracker should be able to renew delegation tokens for the jobs > - > > Key: MAPREDUCE-1430 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1430 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: 1430-BP20-Add.patch, 1430-dd1.patch, 1430-dd2.patch, > 1430-dd3.patch, 1430-dd4-BP20.patch, 1430-dd4.patch, MAPREDUCE-1430-12.patch, > MAPREDUCE-1430-13.patch, MAPREDUCE-1430-14.patch, MAPREDUCE-1430-5.patch, > MAPREDUCE-1430-6.patch, MAPREDUCE-1430-8.patch > > > JobTracker should automatically renew delegation tokens for the jobs it is > currently running. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1493) Authorization for job-history pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1493: - Attachment: MAPREDUCE-1493-20100301.1.txt Update sync'ing with the latest patch at MAPREDUCE-1455. > Authorization for job-history pages > --- > > Key: MAPREDUCE-1493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1493 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security >Reporter: Vinod K V >Assignee: Vinod K V > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1493-20100222.1.txt, > MAPREDUCE-1493-20100225.2.txt, MAPREDUCE-1493-20100226.1.txt, > MAPREDUCE-1493-20100227.2-ydist.txt, MAPREDUCE-1493-20100227.3-ydist.txt, > MAPREDUCE-1493-20100301.1.txt > > > MAPREDUCE-1455 introduces authorization for most of the Map/Reduce jsp pages > and servlets, but left history pages. This JIRA will make sure that > authorization checks are made while accessing job-history pages also. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1455) Authorization for servlets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1455: - Status: Patch Available (was: Open) +1 for the latest patch. > Authorization for servlets > -- > > Key: MAPREDUCE-1455 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1455 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security, tasktracker >Reporter: Devaraj Das >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1455.20S.2.fix.patch, 1455.20S.2.patch, 1455.patch, > 1455.v1.patch, 1455.v2.patch, 1455.v3.patch, 1455.v4.1.patch, > 1455.v4.2.patch, 1455.v4.patch > > > This jira is about building the authorization for servlets (on top of > MAPREDUCE-1307). That is, the JobTracker/TaskTracker runs authorization > checks on web requests based on the configured job permissions. For e.g., if > the job permission is 600, then no one except the authenticated user can look > at the job details via the browser. The authenticated user in the servlet can > be obtained using the HttpServletRequest method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1455) Authorization for servlets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1455: Attachment: 1455.v4.2.patch Attaching patch for trunk. Please review and provide your comments. > Authorization for servlets > -- > > Key: MAPREDUCE-1455 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1455 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security, tasktracker >Reporter: Devaraj Das >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1455.20S.2.fix.patch, 1455.20S.2.patch, 1455.patch, > 1455.v1.patch, 1455.v2.patch, 1455.v3.patch, 1455.v4.1.patch, > 1455.v4.2.patch, 1455.v4.patch > > > This jira is about building the authorization for servlets (on top of > MAPREDUCE-1307). That is, the JobTracker/TaskTracker runs authorization > checks on web requests based on the configured job permissions. For e.g., if > the job permission is 600, then no one except the authenticated user can look > at the job details via the browser. The authenticated user in the servlet can > be obtained using the HttpServletRequest method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839600#action_12839600 ] Vinod K V commented on MAPREDUCE-890: - The patch is not applying over the latest patch at MAPREDUCE-1421 which is a blocker for this JIRA issue. Can you please upload an updated patch? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[
https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-927:
Status: Open (was: Patch Available)
Looked at the patch. Have some comments:
TaskLog.java
- Creation of the {{localFS}} should not be in a static block. In the past
also, we did this and realized it creates a kind of circular initialization of
loggers and results in NPE while creating log objects which can be seen in
task-logs. The current way of creation of {{localFS}} should be retained.
- We can move {{DEFAULT_USER_LOG_RETAIN_HOURS}} to {{TaskLogsCleanupThread}}.
- Shall we rename {{getJobUserLogDir() to a simpler {{getJobDir()}}? And
{{getBaseDir(String)}} to {{getAttemptDir(String)}} to be clear? I think it's
ok like this as {{TaskLog.getJobDir()}} clearly means it is job-dir for logs.
- Also, I think it's high time {{TaskLog.java}} is made
@InterfaceAudience.Private.
TaskLogCleanupThread
- Rename the class to {{TaskLogsMonitor}}, so that we are consistent going
forward with MAPREDUCE-1100.
- Set the audience visibility of the class to private?
- {{threadSleepTime}} is not configurable. May not be a public documented
configuration, but still we need one.
- Constructor: Shouldn't the volume on which the disk-service works be
{{getUserLogsDir()}} instead of {{getBaseLogDir()}}? The correctness is not
lost with the current patch as we are always passing absolute paths to the
disk-service, but I think we should change it anyways. Also can't we simply
construct a local-filesystem here itself, instead of calling
TaskLog.getLocalFileSystem()? This will mostly avoid your changes regarding my
first comment in TaskLog.java above.
- For the sake of correctness, in {{removeOldUserLogs()}}, the job should be
removed only after the deletion of the log file.
- Throughout the class, we should use a clock instead of directly using
{{System.currentMillis()}}. This will better the testing.
- Shouldn't {{clearOldUserLogs()}} be done as part of constructor itself? This
is the pattern that {{MRAsyncDiskService}} uses, for example.
TaskTracker.java
- The {{logscleanup}} thread is not joined/killed in the
{{TaskTracker.close()}}. So, there will be zombie threads in the system on a
re-init and may well interfere with the new thread.
- Shouldn't we be setting 770-user-mapred permissions on userlogs/$jobid as
part of job-localization? Granted 711 is enough for now, but this slightly
deviates from the current security on the directories - we always have the most
secure permissions possible on the files/dirs. If we do this,
TestLocalizationWithLinuxTaskController should also test the permissions of the
joblogdir.
- The following code fragment at +1035 can be moved out of
{{localizeJobFiles()}} into a new method {{initializeJobLogDir()}} and can be
called directly from {{localizeJob()}}.
{code}
taskLogCleanupThread.removeJobFromLogDeletion(jobId);
localizer.initializeJobUserLogDir(jobId);
{code}
- One case is still not handled: TT reinits while the Job is still running.
After re-init, no tasks of the running job arrive at this TT. Retain-hours
after re-init TT removes the job's tasks' logs even though the job is still
running elsewhere. Doing this will need TT to specially communicate with JT and
from what I understand, we are not doing it here. If that is the case, can we
simply add a test for this too in {{TestUserLogCleanup}}?
TestUserLogCleanup
- Use {{TaskTracker.initializeJobLogDir()}} mentioned above in
{{setupJobLogDirectory()}}.
- Once we use a clock in TaskLogCleanupThread,
-- the tests can be modified to use this.
-- we can test what exactly the modified retainTimeStamp is after re-init.
- We should create some attempt dirs also in the joblog dir with appropriate
permissions and verify the proper cleanup.
- Document(javadoc) the two tests?
test-taskcontroller.c
- This needs to be fixed to reflect the new directory hierarchy of the logs.
mapred_tutorial.xml
- Should TaskLogs section be changed to explicitly specify the new directory
hierarchy?
Cancelling the patch for the sake of these changes.
> Cleanup of task-logs should happen in TaskTracker instead of the Child
> --
>
> Key: MAPREDUCE-927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-927
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: security, tasktracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: Amareshwari Sriramadasu
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: patch-927-1.txt, patch-927.txt
>
>
> Task logs' cleanup is being done in Child now. This is undesirable atleast
> for two reasons: 1) failures while cleaning up will affect the user's tasks,
