[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Sarlos updated MAPREDUCE-1306: Attachment: MAPREDUCE-1306-20100308.patch Attaching the updated patch that implements the random seeding option for mumak and rumen. In order to make the simulation deterministic HashSets and HashMaps need to be replaced with different collection classes since they make no guarantees as to the order of iteration. E.g. JobInProgress iterates over the JobTrackers's nodesAtMaxLevel HashSet, the order of this iteration influences the scheduling of non-local maps. Using AspectJ all HashSets and HashMaps are replaced with LinkedHashSets and LinkedHashMaps, whose iteration order is determined by the order of insertions. This solution needs to be revisited if mumak ever becomes multi-threaded. An added functional test verifies deterministic replay by comparing the job history files. [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Sarlos updated MAPREDUCE-1306: Status: Patch Available (was: Open) Submitting patch attached earlier, it passes all test-patch tests on my dev machine. [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1397) NullPointerException observed during task failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842963#action_12842963
]
Hadoop QA commented on MAPREDUCE-1397:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12438253/patch-1397-1.txt
against trunk revision 920250.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/25/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/25/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/25/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/25/console
This message is automatically generated.
NullPointerException observed during task failures
--
Key: MAPREDUCE-1397
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1397
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.20.1
Reporter: Ramya R
Assignee: Amareshwari Sriramadasu
Priority: Minor
Fix For: 0.22.0
Attachments: patch-1397-1.txt, patch-1397.txt
In an environment where many jobs are killed simultaneously, NPEs are
observed in the TT/JT logs when a task fails. The situation is aggravated
when the taskcontroller.cfg is not configured properly. Below is the
exception obtained:
{noformat}
INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_ID:
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:529)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:329)
at
org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:315)
at
org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:146)
at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:109)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:502)
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842972#action_12842972 ] Hadoop QA commented on MAPREDUCE-1420: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435770/MAPREDUCE-1420-v1.patch against trunk revision 920250. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/510/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/510/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/510/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/510/console This message is automatically generated. TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1573) TestStreamingAsDifferentUser fails if run as tt_user
[
https://issues.apache.org/jira/browse/MAPREDUCE-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-1573:
Description:
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't make
the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
was:TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890
didn't make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
TestStreamingAsDifferentUser fails if run as tt_user
Key: MAPREDUCE-1573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1573
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1573.patch
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't
make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1422: -- Attachment: mapreduce-1422-v1.5.4.patch Attaching a new patch with the following changes - added checks to make sure that the job-work-dirs/task-local-dirs contains atleast one path (i.e the false positive concern Hemanth raised) - added extra stray files in task-attempt local dir before calling removeJobFiles() to test the case where the jvm reuse is enabled and attempt dirs are left behind to be cleaned up upon kill-job-action (as suggested by Hemanth) - renamed the _TaskTracker#buildTaskControllerPathDeletionContexts()_ to _TaskTracker#buildTaskControllerTaskPathDeletionContexts()_ (as suggested by Ravi) test-patch passed. All ant tests passed. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1573) TestStreamingAsDifferentUser fails if run as tt_user
[
https://issues.apache.org/jira/browse/MAPREDUCE-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842981#action_12842981
]
Vinod K V commented on MAPREDUCE-1573:
--
Patch looks good. +1.
TestStreamingAsDifferentUser fails if run as tt_user
Key: MAPREDUCE-1573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1573
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, test
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1573.patch
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't
make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1573) TestStreamingAsDifferentUser fails if run as tt_user
[
https://issues.apache.org/jira/browse/MAPREDUCE-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-1573:
-
Component/s: test
task-controller
Hadoop Flags: [Reviewed]
TestStreamingAsDifferentUser fails if run as tt_user
Key: MAPREDUCE-1573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1573
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, test
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1573.patch
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't
make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1573) TestStreamingAsDifferentUser fails if run as tt_user
[
https://issues.apache.org/jira/browse/MAPREDUCE-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-1573:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just checked this in. Thanks Ravi!
TestStreamingAsDifferentUser fails if run as tt_user
Key: MAPREDUCE-1573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1573
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, test
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1573.patch
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't
make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1573) TestStreamingAsDifferentUser fails if run as tt_user
[
https://issues.apache.org/jira/browse/MAPREDUCE-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842995#action_12842995
]
Hudson commented on MAPREDUCE-1573:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #268 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/268/])
. TestStreamingAsDifferentUser fails if run as tt_user. Contributed by Ravi
Gummadi.
TestStreamingAsDifferentUser fails if run as tt_user
Key: MAPREDUCE-1573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1573
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, test
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1573.patch
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't
make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842996#action_12842996 ] Hemanth Yamijala commented on MAPREDUCE-1415: - Looks good overall. Very minor suggestions: - In PipeMapRed.configure, can we use File.setExecutable(true) instead of FileUtil.chmod ? With FileUtil.chmod, I suppose an exception will be thrown if there's an invalid path passed. To maintain that semantics, we probably need to check the return value of File.setExecutable and throw a wrapped IOException when that happens. - In testStreaming, we can use TEST_DIR.toAbsolutePath() as the value of stream.tmpdir. - Very minor - in testStreamingWithDistCache, I suggest we use a variable to store the file name of the input script, test.sh. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1580) Tasklog's job-acl file should be one for job
Tasklog's job-acl file should be one for job Key: MAPREDUCE-1580 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1580 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.22.0 Reporter: Amareshwari Sriramadasu Fix For: 0.22.0 Currently, job-acl file is created in every attempt directory of user-logs. MAPREDUCE-927 puts jobid/attemptid hierarchy for user-log directories. job-acl file can be moved to jobid directory and can be localized along with the log directory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843016#action_12843016 ] Hadoop QA commented on MAPREDUCE-1306: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438267/MAPREDUCE-1306-20100308.patch against trunk revision 920250. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/26/console This message is automatically generated. [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1582) JobClient#monitorAndPrintJob() should use logger for logging taskDiagnostics
JobClient#monitorAndPrintJob() should use logger for logging taskDiagnostics Key: MAPREDUCE-1582 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1582 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission Reporter: Johannes Zillmann Priority: Minor In JobClient#monitorAndPrintJob() (around line 1324 in version 0.20.1) the exception of tasktrackers are printed via system.err. Using LOG.error() instead would make capturing the logs easier ! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-927: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Amareshwari! Cleanup of task-logs should happen in TaskTracker instead of the Child -- Key: MAPREDUCE-927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security, tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Priority: Blocker Fix For: 0.22.0 Attachments: patch-927-1.txt, patch-927-2.txt, patch-927-3.txt, patch-927-4.txt, patch-927.txt Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Status: Patch Available (was: Open) TestStreamingAsDifferentUser passes with both tt_user and some-other user. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415-2.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Status: Open (was: Patch Available) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1105) CapacityScheduler: It should be possible to set queue hard-limit beyond it's actual capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-1105: Attachment: MAPREDUCE-1105_apache0202.txt CapacityScheduler: It should be possible to set queue hard-limit beyond it's actual capacity Key: MAPREDUCE-1105 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1105 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.21.0 Reporter: Arun C Murthy Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1105-21-1.patch, MAPRED-1105-21-2.patch, MAPRED-1105-21-3.patch, MAPRED-1105-21-3.patch, MAPREDUCE-1105-version20-2.patch, MAPREDUCE-1105-version20.patch.txt, MAPREDUCE-1105-yahoo-version20-3.patch, MAPREDUCE-1105-yahoo-version20-4.patch, MAPREDUCE-1105-yahoo-version20-5.patch, MAPREDUCE-1105_apache0202.txt Currently the CS caps a queue's capacity to it's actual capacity if a hard-limit is specified to be greater than it's actual capacity. We should allow the queue to go upto the hard-limit if specified. Also, I propose we change the hard-limit unit to be percentage rather than #slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1569) Mock Contexts Configurations
[ https://issues.apache.org/jira/browse/MAPREDUCE-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris White updated MAPREDUCE-1569: --- Status: In Progress (was: Patch Available) Attempting to restart Hudson job Mock Contexts Configurations -- Key: MAPREDUCE-1569 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1569 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mrunit Reporter: Chris White Priority: Minor Attachments: MAPREDUCE-1569.patch Original Estimate: 1h Remaining Estimate: 1h Currently the library creates a new Configuration object in the MockMapContext and MocKReduceContext constructors, rather than allowing the developer to configure and pass their own -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Attachment: patch-1415-2.txt Patch with comments incorporated. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415-2.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1422: -- Attachment: mapreduce-1422-v1.5.5.patch MAPREDUCE-927 got committed after I uploaded the patch. Attaching a new patch incorporating changes from the merge. test-patch passed. This is only a test-case change and tested the test-case separately. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843195#action_12843195 ] Hudson commented on MAPREDUCE-927: -- Integrated in Hadoop-Mapreduce-trunk #253 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/253/]) . Cleanup of task-logs should happen in TaskTracker instead of the Child. Contributed by Amareshwari Sriramadasu. Cleanup of task-logs should happen in TaskTracker instead of the Child -- Key: MAPREDUCE-927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security, tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-927-1.txt, patch-927-2.txt, patch-927-3.txt, patch-927-4.txt, patch-927.txt Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1573) TestStreamingAsDifferentUser fails if run as tt_user
[
https://issues.apache.org/jira/browse/MAPREDUCE-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843196#action_12843196
]
Hudson commented on MAPREDUCE-1573:
---
Integrated in Hadoop-Mapreduce-trunk #253 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/253/])
. TestStreamingAsDifferentUser fails if run as tt_user. Contributed by Ravi
Gummadi.
TestStreamingAsDifferentUser fails if run as tt_user
Key: MAPREDUCE-1573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1573
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task-controller, test
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1573.patch
TestStreamingAsDifferentUser fails if run as tt_user. MAPREDUCE-890 didn't
make the necessary changes needed for the newly added testcase in
TestStreamignAsDifferentUser.
{code}
Testcase: testStreamingWithDistCache took 21.228 sec
FAILED
Path /tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws---
junit.framework.AssertionFailedError: Path
/tmp/hadoop-gravi/mapred/local/0_0/taskTracker/gravi/distcache has the
permissions drwxrws--- instead of the expected dr-xrws--- at
org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:292)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnDir(ClusterWithLinuxTaskController.java:440)
at
org.apache.hadoop.mapred.ClusterWithLinuxTaskController.checkPermissionsOnPrivateDistCache(ClusterWithLinuxTaskController.java:354)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:157)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser$2.run(TestStreamingAsDifferentUser.java:120)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
at
org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreamingWithDistCache(TestStreamingAsDifferentUser.java:120)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1583) teragen t parameter may be overflowing
teragen t parameter may be overflowing
Key: MAPREDUCE-1583
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1583
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 0.22.0
Reporter: Steve Loughran
Priority: Minor
My IDE is warning me that the logic in TeraGen.java to apply a T value is
overflowing:
{code}
case 't':
base *= 1000 * 1000 * 1000 * 1000; //here
break;
{code}
Perhaps the IDE thinks/knows that the compiler will multiply the four integers
to produce a compile time constant, so one of them needs to be marked as 1000L
to stop the size of a terabyte being much less than expected, with consequences
for sort times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1584) analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps
analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps - Key: MAPREDUCE-1584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1584 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.22.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.22.0 analysejobhistory.jsp returns without proceeding to 'analysing reduces' if the number of maps in the job is 0. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1578) HadoopArchives.java should not use HarFileSystem.VERSION
[ https://issues.apache.org/jira/browse/MAPREDUCE-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843224#action_12843224 ] Mahadev konar commented on MAPREDUCE-1578: -- +1 the patch looks good. I have been thinking about moving out HarFileSystem to mapreduce and get it into contrib or some seperate tools jar so that its not part of the core release. We might want to do that ASAP. We can commit this for now, but moving it out of core is very essential for folks to use Har! HadoopArchives.java should not use HarFileSystem.VERSION Key: MAPREDUCE-1578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1578 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Attachments: MAPREDUCE-1578.patch If we upgrade the protocol on HarFileSystem, HadoopArchives might generate an old archive and assign the new version number to it. This should be fixed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843223#action_12843223 ] Hemanth Yamijala commented on MAPREDUCE-1422: - The latest patch looks good to me. +1. Note though that: bq. added checks to make sure that the job-work-dirs/task-local-dirs contains atleast one path This is not checking the existence of paths, the returned array is simply a list of path objects created with no bearing on the physical directory / file creation. However, I think this is still fine because calling localizeJob ensures work directory is created this is tested in testJobLocalization(). Likewise, calling localizeTask ensures attempt directory is created.. this is tested in testTaskLocalization. So, I think it is OK even to not check these conditions in the new test method. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1422: Status: Patch Available (was: Open) Trying Hudson. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1578) HadoopArchives.java should not use HarFileSystem.VERSION
[ https://issues.apache.org/jira/browse/MAPREDUCE-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-1578: - Fix Version/s: 0.22.0 HadoopArchives.java should not use HarFileSystem.VERSION Key: MAPREDUCE-1578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1578 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1578.patch If we upgrade the protocol on HarFileSystem, HadoopArchives might generate an old archive and assign the new version number to it. This should be fixed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843235#action_12843235 ] Hemanth Yamijala commented on MAPREDUCE-1415: - Looking at one of the failed tests, it appears that we have test cases that use default commands like 'xargs' as the mapper, but don't specify the full path. Hence the code tries to set permissions for it in the current working directory and fails (because the file does not exist). I thought FileUtil.fullyDelete actually throws an exception if there's a command failure, but I see in code that we actually ignore exceptions explicitly. I assume the failure should have been seen before as well - at some debug log level. So, one option could be to not check the return code of setPermissions. Another option is to change the tests to work with full paths. However, this is inherently system dependent (as xargs could potentially be under multiple directories depending on the system). With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415-2.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1584) analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1584: Attachment: 1584.patch Attaching patch fixing analysejobhistory.jsp to display analysis of reduce tasks of job even when number of maps is 0. Please review and provide your comments. analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps - Key: MAPREDUCE-1584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1584 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.22.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.22.0 Attachments: 1584.patch analysejobhistory.jsp returns without proceeding to 'analysing reduces' if the number of maps in the job is 0. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1584) analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1584: Status: Patch Available (was: Open) analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps - Key: MAPREDUCE-1584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1584 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.22.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.22.0 Attachments: 1584.patch analysejobhistory.jsp returns without proceeding to 'analysing reduces' if the number of maps in the job is 0. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1578) HadoopArchives.java should not use HarFileSystem.VERSION
[ https://issues.apache.org/jira/browse/MAPREDUCE-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-1578: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I have committed this. Thanks, Rodrigo! HadoopArchives.java should not use HarFileSystem.VERSION Key: MAPREDUCE-1578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1578 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1578.patch If we upgrade the protocol on HarFileSystem, HadoopArchives might generate an old archive and assign the new version number to it. This should be fixed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1579) archive: check and possibly replace the space charater in paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843260#action_12843260 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1579: --- I suggest to have two config properties for the version 1 HadoopArchives. - har.space.replace.enable = true|false - har.space.replacement = REPLACEMENT_STRING If space replacement is not enabled, an exception will be thrown if there are spaces in the paths. archive: check and possibly replace the space charater in paths --- Key: MAPREDUCE-1579 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1579 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Tsz Wo (Nicholas), SZE Assignee: Mahadev konar Since the space character is used as a separator in the index files, it won't work if there are spaces in the path (see also HADOOP-6591). The archive tools should # detect if there are spaces in the paths and # provide an option to replace it with some other characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1579) archive: check and possibly replace the space charater in paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843262#action_12843262 ] Rodrigo Schmidt commented on MAPREDUCE-1579: +1 to Nicholas' suggestion. archive: check and possibly replace the space charater in paths --- Key: MAPREDUCE-1579 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1579 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Tsz Wo (Nicholas), SZE Assignee: Mahadev konar Since the space character is used as a separator in the index files, it won't work if there are spaces in the path (see also HADOOP-6591). The archive tools should # detect if there are spaces in the paths and # provide an option to replace it with some other characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
[ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843268#action_12843268 ] Hudson commented on MAPREDUCE-927: -- Integrated in Hadoop-Mapreduce-trunk-Commit #270 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/270/]) Cleanup of task-logs should happen in TaskTracker instead of the Child -- Key: MAPREDUCE-927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security, tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-927-1.txt, patch-927-2.txt, patch-927-3.txt, patch-927-4.txt, patch-927.txt Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1578) HadoopArchives.java should not use HarFileSystem.VERSION
[ https://issues.apache.org/jira/browse/MAPREDUCE-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843269#action_12843269 ] Hudson commented on MAPREDUCE-1578: --- Integrated in Hadoop-Mapreduce-trunk-Commit #270 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/270/]) . Decouple HadoopArchives vesrion from HarFileSystem version. Contributed by Rodrigo Schmidt HadoopArchives.java should not use HarFileSystem.VERSION Key: MAPREDUCE-1578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1578 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1578.patch If we upgrade the protocol on HarFileSystem, HadoopArchives might generate an old archive and assign the new version number to it. This should be fixed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1584) analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843315#action_12843315 ] Hadoop QA commented on MAPREDUCE-1584: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438308/1584.patch against trunk revision 920793. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/27/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/27/console This message is automatically generated. analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps - Key: MAPREDUCE-1584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1584 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.22.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.22.0 Attachments: 1584.patch analysejobhistory.jsp returns without proceeding to 'analysing reduces' if the number of maps in the job is 0. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843321#action_12843321 ] Hadoop QA commented on MAPREDUCE-1422: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438295/mapreduce-1422-v1.5.5.patch against trunk revision 920793. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/512/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/512/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/512/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/512/console This message is automatically generated. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1403: --- Attachment: mr-1403-trunk-v2.patch Discussed with Chris on item 2 and 3, Incorporated the rest of the suggestions. Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1403: --- Status: Patch Available (was: Open) Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843338#action_12843338 ] Hadoop QA commented on MAPREDUCE-1403: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438333/mr-1403-trunk-v2.patch against trunk revision 921069. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/513/console This message is automatically generated. Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843344#action_12843344 ] Scott Chen commented on MAPREDUCE-1420: --- Thanks for the help, Vinod. It is strange that there are many failed tests. The patch only changes one unit tests. There is no dependency between this one and those unit tests. I will investigate this. TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1420: -- Status: Patch Available (was: Open) TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1403: --- Attachment: mr-1403-trunk-v3.patch Rebased the patch against trunk. Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch, mr-1403-trunk-v3.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1403: - Status: Open (was: Patch Available) Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch, mr-1403-trunk-v3.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1403: - Status: Patch Available (was: Open) Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch, mr-1403-trunk-v3.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Sarlos updated MAPREDUCE-1306: Status: Open (was: Patch Available) Canceling patch so that I can resubmit it [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Sarlos updated MAPREDUCE-1306: Status: Patch Available (was: Open) Resubmitting latest patch to Hudson as failing test runs fine locally. [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843380#action_12843380 ] Rodrigo Schmidt commented on MAPREDUCE-1585: I've uploaded a patch, but it is dependent on the code I proposed for HDFS-6591. Create Hadoop Archives version 2 with filenames URL-encoded --- Key: MAPREDUCE-1585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Attachments: MAPREDUCE-1585.patch Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HDFS-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HDFS-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt updated MAPREDUCE-1585: --- Attachment: MAPREDUCE-1585.patch Create Hadoop Archives version 2 with filenames URL-encoded --- Key: MAPREDUCE-1585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Attachments: MAPREDUCE-1585.patch Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HDFS-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HDFS-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843383#action_12843383 ] Rodrigo Schmidt commented on MAPREDUCE-1585: I mean HADOOP-6591 Create Hadoop Archives version 2 with filenames URL-encoded --- Key: MAPREDUCE-1585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Attachments: MAPREDUCE-1585.patch Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HDFS-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HDFS-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt updated MAPREDUCE-1585: --- Description: Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HADOOP-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HADOOP-6591 was: Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HDFS-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HDFS-6591 Create Hadoop Archives version 2 with filenames URL-encoded --- Key: MAPREDUCE-1585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Attachments: MAPREDUCE-1585.patch Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HADOOP-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HADOOP-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1420: - Status: Open (was: Patch Available) TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1420: - Status: Patch Available (was: Open) The failures are the usual suspects - missing common classes similar to failures in MAPREDUCE-890, MAPREDUCE-927 etc., a wierd scenario which i could never explain, and HADOOP-6528. Resubmitting to Hudson again with the hope of a test-pass this time. TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Status: Open (was: Patch Available) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415-2.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1418) LinuxTaskController binary misses validation of arguments passed for relative components in some cases.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala reassigned MAPREDUCE-1418:
---
Assignee: Hemanth Yamijala
LinuxTaskController binary misses validation of arguments passed for relative
components in some cases.
---
Key: MAPREDUCE-1418
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1418
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security, tasktracker
Reporter: Vinod K V
Assignee: Hemanth Yamijala
The function {{int check_path_for_relative_components(char * path)}} should
be used to validate the absence of relative components before any operation
is done on those paths. This is missed in all the {{initialize*()}}
functions, as Hemanth pointed out offline.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1418) LinuxTaskController binary misses validation of arguments passed for relative components in some cases.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843432#action_12843432
]
Hemanth Yamijala commented on MAPREDUCE-1418:
-
The following functions in task-controller need modification:
- initialize_user
- initialize_task
- initialize_job
- initialize_distributed_cache_file
I will try and add unit tests in appropriate LinuxTaskController related test
classes that call the taskcontroller with relative paths in arguments and
verify the right error occurs.
LinuxTaskController binary misses validation of arguments passed for relative
components in some cases.
---
Key: MAPREDUCE-1418
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1418
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security, tasktracker
Reporter: Vinod K V
Assignee: Hemanth Yamijala
The function {{int check_path_for_relative_components(char * path)}} should
be used to validate the absence of relative components before any operation
is done on those paths. This is missed in all the {{initialize*()}}
functions, as Hemanth pointed out offline.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843433#action_12843433 ] Hadoop QA commented on MAPREDUCE-1420: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435770/MAPREDUCE-1420-v1.patch against trunk revision 921069. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/514/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/514/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/514/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/514/console This message is automatically generated. TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843434#action_12843434 ] Hadoop QA commented on MAPREDUCE-1403: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438342/mr-1403-trunk-v3.patch against trunk revision 921069. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/28/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/28/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/28/console This message is automatically generated. Save file-sizes of each of the artifacts in DistributedCache in the JobConf --- Key: MAPREDUCE-1403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.22.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Attachments: MAPREDUCE-1403_yhadoop20-1.patch, MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch, MR-1403-trunk-1.patch, mr-1403-trunk-v2.patch, mr-1403-trunk-v3.patch It would be a useful metric to collect... potentially GridMix could use it to emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1584) analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps
[
https://issues.apache.org/jira/browse/MAPREDUCE-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843437#action_12843437
]
Amar Kamat commented on MAPREDUCE-1584:
---
Looking at the code, it looks like there are 3 states for the task : MAP,
SHUFFLE and REDUCE. Also for each of these states, following is printed : Best.
Avg and Last. Can we factor out the code? Something like
{code}
printTasks(out, TaskType, tasks[], comparator) {
newtasks[] = Arrays.sort(tasks, comparator);
out.print(h1Printing best performing + TaskType.toString() + task/h1);
// print best
// print avg
// print worst
}
TaskType can be {MAP, SHUFFLE, REDUCE}
{code}
If yes, we can do something like
{code}
if (maps.length 0) {
printTasks(MAP, maps, mapComp);
}
if (reduces.length 0) {
printTasks(SHUFFLE, reduces, shuffleComp);
printTasks(REDUCE, reduces, redComp);
}
{code}
Ravi, can you please check if this can be done easily?
analysejobhistory.jsp doesn't analyse reduces of jobs with 0 maps
-
Key: MAPREDUCE-1584
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1584
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1584.patch
analysejobhistory.jsp returns without proceeding to 'analysing reduces' if
the number of maps in the job is 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1553) mapred.userlog.retain.hours is improperly renamed in MAPREDUCE-849
[ https://issues.apache.org/jira/browse/MAPREDUCE-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843440#action_12843440 ] Amareshwari Sriramadasu commented on MAPREDUCE-1553: The patch is applicable only for branch 0.21. The same change went into trunk as part of MAPREDUCE-927. mapred.userlog.retain.hours is improperly renamed in MAPREDUCE-849 -- Key: MAPREDUCE-1553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1553 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 0.21.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Priority: Blocker Fix For: 0.21.0 Attachments: patch-1553.txt mapred.userlog.retain.hours is renamed as mapred.task.userlog.retain.hours in JobContext. But, in mapred-default, it is mapreduce.task.userlog.retain.hours. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843441#action_12843441 ] Hemanth Yamijala commented on MAPREDUCE-1422: - All LinuxTaskController tests also passed. This is good for commit. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1422: Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this to trunk. Thanks, Amar ! Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Fix For: 0.22.0 Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Attachment: MR890.20S.patch Attaching patch for earlier version of hadoop(includes the fix of MAPREDUCE-1573 also). Not for commit here. After HADOOP-4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Karam Singh Assignee: Ravi Gummadi Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843446#action_12843446 ] Hudson commented on MAPREDUCE-1422: --- Integrated in Hadoop-Mapreduce-trunk-Commit #271 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/271/]) . Fix cleanup of localized job directory to work if files with non-deletable permissions are created within it. Contributed by Amar Kamat. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Fix For: 0.22.0 Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1586) TestWebUIAuthorization fails sometimes
TestWebUIAuthorization fails sometimes
--
Key: MAPREDUCE-1586
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1586
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Ravi Gummadi
Testcase testAuthorizationForJobHistoryPages() failed once on my local machine
with
{code}
[junit] 2010-03-10 10:16:42,230 INFO mapred.TestWebUIAuthorization
(TestWebUIAuthorization.java:getHttpStatusCode(83)) - Accessing
http://localhost:46846/jobdetailshistory.jsp?logFile= as user user1
[junit] 22751 [31606...@qtp3-0] ERROR org.mortbay.log -
/jobdetailshistory.jsp
[junit] java.lang.IllegalArgumentException: Can not create a Path from an
empty string
[junit] at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
[junit] at org.apache.hadoop.fs.Path.init(Path.java:90)
[junit] at
org.apache.hadoop.mapred.jobdetailshistory_jsp._jspService(jobdetailshistory_jsp.java:90)
{code}
I guess the testcase needs to wait for retiring of job instead of only waiting
for completion of job before accessing history related jsps
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843449#action_12843449 ] Scott Chen commented on MAPREDUCE-1420: --- Test succeed after resubmitted TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1422: -- Release Note: Introduced enableJobForCleanup() api in TaskController. This api enables deletion of stray files (with no write permissions for task-tracker) from job's work dir. Note that the behavior is similar to TaskController#enableTaskForCleanup() except the path on which the 'chmod' is done is the job's work dir. Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Fix For: 0.22.0 Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch, mapreduce-1422-v1.5.4.patch, mapreduce-1422-v1.5.5.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1306: - Attachment: MAPREDUCE-1306-20100308-hong.patch [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1306) [MUMAK] Randomize the arrival of heartbeat responses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843455#action_12843455 ] Hong Tang commented on MAPREDUCE-1306: -- I think we should avoid using conf objects to pass the seeds from SimulatorEngine to ZombieJobStoryProducer. This creates a hidden dependency from SimulatorEngine to ZJSP, and thus is harder to maintain in the long run. I revised your patch such that SE passes a seed to SimulatorJobStoryProducer (SJSP) which in turn passes down to ZJSP. [MUMAK] Randomize the arrival of heartbeat responses Key: MAPREDUCE-1306 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1306 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Tamas Sarlos Assignee: Tamas Sarlos Fix For: 0.21.0, 0.22.0 Attachments: MAPREDUCE-1306-20100108.patch, MAPREDUCE-1306-20100308-hong.patch, MAPREDUCE-1306-20100308.patch We propose to make the following changes to mumak, MAPREDUCE-728 - make the timing of heartbeat responses more realistic by adding an option to randomly perturb them - randomize the startup time of task trackers in a fixed interval - remove 2 magic constants from SimulatorEngine and make sure that the first job is submitted only after the entire cluster is up and running -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1542) Deprecate mapred.permissions.supergroup in favor of hadoop.cluster.administrators
[
https://issues.apache.org/jira/browse/MAPREDUCE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12843458#action_12843458
]
Ravi Gummadi commented on MAPREDUCE-1542:
-
Configuration without default resources does not load *site.xml files also. So
needs to change the logic in buildAdminsACL().
Deprecate mapred.permissions.supergroup in favor of
hadoop.cluster.administrators
-
Key: MAPREDUCE-1542
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1542
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Reporter: Vinod K V
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1542.patch, 1542.v1.patch
HADOOP-6568 added the configuration {{hadoop.cluster.administrators}} through
which admins can configure who the superusers/supergroups for the cluster
are. MAPREDUCE itself already has {{mapred.permissions.supergroup}} (which is
just a single group). As agreed upon at HADOOP-6568, this should be
deprecated in favor of {{hadoop.cluster.administrators}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1566) Need to add a mechanism to import tokens and secrets into a submitted job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-1566: --- Attachment: mr-1566-1.1.patch This patch is a bugfix on top of the earlier patch. Need to add a mechanism to import tokens and secrets into a submitted job. -- Key: MAPREDUCE-1566 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1566 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.22.0 Attachments: mr-1566-1.1.patch, mr-1566-1.patch We need to include tokens and secrets into a submitted job. I propose adding a configuration attribute that when pointed at a token storage file will include the tokens and secrets from that token storage file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
