[jira] Commented: (MAPREDUCE-1562) TestBadRecords fails sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841746#action_12841746
]
Vinod K V commented on MAPREDUCE-1562:
--
Two things I immediately can identify:
- One TT crashed because of jetty port problem, possibly related to HADOOP-6528
{code}
[exec] [junit] 2010-03-04 09:40:24,493 ERROR mapred.TaskTracker
(TaskTracker.java:offerService(1460)) - Caught exception: java.io.IOException:
Jetty problem. Jetty didn't bind to a valid port
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.checkJettyPort(TaskTracker.java:1276)
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1438)
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2256)
[exec] [junit] at
org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:228)
[exec] [junit] at java.lang.Thread.run(Thread.java:619)
{code}
I think we should fail *any* test immediately if MiniMRCluster fails to start
even a single TT.
- Two, I see this 15 min wait (!!)
{code}
else if(MAPPER_BAD_RECORDS.get(2).equals(str)) {
try {
LOG.warn(MAP Encountered BAD record);
Thread.sleep(15*60*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
{code}
Not sure why this is not causing the test to fail all the time.
TestBadRecords fails sometimes
--
Key: MAPREDUCE-1562
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1562
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Vinod K V
Fix For: 0.22.0
Attachments: consoleText
TestBadRecords.testMapRed fails sometimes. One instance of this was seen by
Hudson while testing MAPREDUCE-890:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841747#action_12841747 ] Vinod K V commented on MAPREDUCE-890: - The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why? Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed. After HADOOP-4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Karam Singh Assignee: Ravi Gummadi Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1527) QueueManager should issue warning if mapred-queues.xml is skipped.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1527: - Resolution: Fixed Fix Version/s: 0.22.0 Status: Resolved (was: Patch Available) The test failure is unrelated, this is a trivial change to a LOG.warn. I just committed this. Thanks Hong! QueueManager should issue warning if mapred-queues.xml is skipped. -- Key: MAPREDUCE-1527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1527 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Hong Tang Assignee: Hong Tang Fix For: 0.22.0 Attachments: mr-1527-20100224-2.patch, mr-1527-20100224.patch MAPREDUCE-861 added support for hierarchical queues and used a new file mapred-queues.xml to configure queues. But the QueueManager will silently skip the file if property mapred.queue.names is defined in Configuration. This tripped us when we are testing MAPREDUCE-1235 and copied configuration files from a Hadoop 0.20 cluster. I suggest QueueManager issue a friendly warning if both of the following are true: (1) mapred.queue.names exists in Configuration; (2) mapred-queues.xml is found in the classpath. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1563) Task diagnostic info would get missed sometimes.
Task diagnostic info would get missed sometimes. Key: MAPREDUCE-1563 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1563 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Amareshwari Sriramadasu Fix For: 0.22.0 TaskStatus's diagnostic info is reset in TaskTracker.cloneAndResetRunningTaskStatuses() method while constructing TaskTrackerStatus to send the heartbeat. It is also reset after receiving the response in TaskTracker.transmitHeartbeat() method. If any task sets diagnostic info between both the clearStatus() calls, the info is lost. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1560) Better diagnostic message for tasks killed for going over vmem limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V resolved MAPREDUCE-1560. -- Resolution: Invalid We already have this feature. It is just that because of a bug in the diagnostics framework, we don't always record the information. Head to MAPREDUCE-1563 for the bug in question. Resolving this as invalid. Better diagnostic message for tasks killed for going over vmem limit Key: MAPREDUCE-1560 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1560 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Currently the user has no indication of his tasks getting killed due to vmem limit, the only way to know is by looking at TT logs. We should get the TT to insert a diagnostic string for the task to indicate this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1560) Better diagnostic message for tasks killed for going over vmem limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841757#action_12841757 ] Arun C Murthy commented on MAPREDUCE-1560: -- Thanks Vinod. Better diagnostic message for tasks killed for going over vmem limit Key: MAPREDUCE-1560 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1560 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 Currently the user has no indication of his tasks getting killed due to vmem limit, the only way to know is by looking at TT logs. We should get the TT to insert a diagnostic string for the task to indicate this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1455) Authorization for servlets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1455: Attachment: 1455.20S.2.fix1.patch Patch fixing testcase for earlier version of hadoop. Not for commit here. Authorization for servlets -- Key: MAPREDUCE-1455 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1455 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobtracker, security, tasktracker Reporter: Devaraj Das Assignee: Ravi Gummadi Fix For: 0.22.0 Attachments: 1455.20S.2.fix.patch, 1455.20S.2.fix1.patch, 1455.20S.2.patch, 1455.patch, 1455.v1.patch, 1455.v2.patch, 1455.v3.patch, 1455.v4.1.patch, 1455.v4.2.patch, 1455.v4.patch This jira is about building the authorization for servlets (on top of MAPREDUCE-1307). That is, the JobTracker/TaskTracker runs authorization checks on web requests based on the configured job permissions. For e.g., if the job permission is 600, then no one except the authenticated user can look at the job details via the browser. The authenticated user in the servlet can be obtained using the HttpServletRequest method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1518) On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It would be better if it didn't.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-1518: Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Rodrigo! On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It would be better if it didn't. - Key: MAPREDUCE-1518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1518 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/raid Environment: On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It runs okay because the directory is not empty and trying to delete it non-recursively fails, but such failure messages only polute the log file. My proposal is the following: If recursePurge is checking a directory, it should call itself recursively. If it's checking a file, it should do the deletion check. Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1518.0.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Status: Open (was: Patch Available) After HADOOP-4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Karam Singh Assignee: Ravi Gummadi Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.patch Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1562) TestBadRecords fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841760#action_12841760 ] Amareshwari Sriramadasu commented on MAPREDUCE-1562: bq. I think we should fail any test immediately if MiniMRCluster fails to start even a single TT. MAPREDUCE-1366 is open for this. TestBadRecords fails sometimes -- Key: MAPREDUCE-1562 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1562 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Vinod K V Fix For: 0.22.0 Attachments: consoleText TestBadRecords.testMapRed fails sometimes. One instance of this was seen by Hudson while testing MAPREDUCE-890: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1527) QueueManager should issue warning if mapred-queues.xml is skipped.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841761#action_12841761 ] Hudson commented on MAPREDUCE-1527: --- Integrated in Hadoop-Mapreduce-trunk-Commit #263 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/263/]) . Better warning logged when mapred.queue.names is overshadowed by mapred-queues.xml. Contributed by Hong Tang. QueueManager should issue warning if mapred-queues.xml is skipped. -- Key: MAPREDUCE-1527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1527 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Hong Tang Assignee: Hong Tang Fix For: 0.22.0 Attachments: mr-1527-20100224-2.patch, mr-1527-20100224.patch MAPREDUCE-861 added support for hierarchical queues and used a new file mapred-queues.xml to configure queues. But the QueueManager will silently skip the file if property mapred.queue.names is defined in Configuration. This tripped us when we are testing MAPREDUCE-1235 and copied configuration files from a Hadoop 0.20 cluster. I suggest QueueManager issue a friendly warning if both of the following are true: (1) mapred.queue.names exists in Configuration; (2) mapred-queues.xml is found in the classpath. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Attachment: MR890.v1.2.patch Syncing with the latest trunk(as MAPREDUCE-1435 went in). After HADOOP-4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Karam Singh Assignee: Ravi Gummadi Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Status: Patch Available (was: Open) Allowing it to go through Hudson After HADOOP-4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Karam Singh Assignee: Ravi Gummadi Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841766#action_12841766 ] Amar Kamat commented on MAPREDUCE-1422: --- Hemanth, I am confused w.r.t point #3 and #4. I added test-code in TestTaskTrackerLocalization for testing job-work-dir cleanup because there is a call to _tracker.removeJobFiles()_. Now TestLocalizationWithLinuxTaskController (that extends TestTaskTrackerLocalization) will also test the job-work-dir cleanup code with LinuxTaskController. As you are suggesting, if we move this job cleanup test-code into a separate testcase, then we need TestJobLocalDirCleanupWithLinuxTaskController, no? Another way to do this would be to introduce a new testcase (say TestJobDirCleanup) as you suggested which will test the job-dir and task-dirs upon job completion. Also add a testcase (say TestJobDirCleanupWithLinuxTaskController) extending TestJobDirCleanup but configured with LinuxTaskController. Thoughts? Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always - Key: MAPREDUCE-1422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Ravi Gummadi Assignee: Amar Kamat Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up of these directories sothat cleanup will be succeeded even if user had created files/dirs under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions and user can create files/dirs under job-work-dir with non-writable permissions, Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1421) LinuxTaskController tests failing on trunk after the commit of MAPREDUCE-1385
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1421: --- Attachment: patch-1421-1-ydist.txt Patch for Yahoo! distribution. One file was missed in earlier patch. LinuxTaskController tests failing on trunk after the commit of MAPREDUCE-1385 - Key: MAPREDUCE-1421 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller, tasktracker, test Affects Versions: 0.22.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1421-1-ydist.txt, patch-1421-1.txt, patch-1421-2.txt, patch-1421-ydist.txt, patch-1421.txt, TestJobExecutionAsDifferentUser.patch The following tests fail, in particular: - TestDebugScriptWithLinuxTaskController - TestJobExecutionAsDifferentUser - TestPipesAsDifferentUser - TestKillSubProcessesWithLinuxTaskController -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1518) On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It would be better if it didn't.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841773#action_12841773 ] Hudson commented on MAPREDUCE-1518: --- Integrated in Hadoop-Mapreduce-trunk-Commit #264 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/264/]) . RaidNode does not run the deletion check on the directory that stores the parity files. (Rodrigo Schmidt via dhruba) On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It would be better if it didn't. - Key: MAPREDUCE-1518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1518 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/raid Environment: On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It runs okay because the directory is not empty and trying to delete it non-recursively fails, but such failure messages only polute the log file. My proposal is the following: If recursePurge is checking a directory, it should call itself recursively. If it's checking a file, it should do the deletion check. Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1518.0.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-653) distcp can support bandwidth limiting
[ https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841778#action_12841778 ] Ravi Gummadi commented on MAPREDUCE-653: We need to experiment and check if this way of asking JobTracker for counters from each map task will be a considerable overhead on JobTracker(and would this make JobTracker heavily loaded serving these many requests, with lot more new connections to JobTracker). distcp can support bandwidth limiting - Key: MAPREDUCE-653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-653 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distcp Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch distcp should support an option for user to specify the bandwidth limit for the distcp job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1403:
-
Status: Open (was: Patch Available)
* The patch includes a whitespace change to {{Job}}
* Can you explain the addition of {{getConfiguration}} to the {{RunningJob}}
interface? Is the relevant copy in {{JobContextImpl}}?
* Please retain javadoc for {{CACHE_FILES_SIZES}} and {{CACHE_ARCHIVES_SIZES}},
rather than code comments
* javadoc referring to classes (as
{{TrackerDistributedCacheManager::getFileStatus}}) should probably use
{...@link org.class.name}}} instead of {{codename/code}}
* The actual/expected in the error message for {{assertEquals}} is redundant
Save file-sizes of each of the artifacts in DistributedCache in the JobConf
---
Key: MAPREDUCE-1403
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.22.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.22.0
Attachments: MAPREDUCE-1403_yhadoop20-1.patch,
MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch,
MR-1403-trunk-1.patch
It would be a useful metric to collect... potentially GridMix could use it to
emulate jobs which use the DistributedCache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1466:
-
Status: Open (was: Patch Available)
* Please retain javadoc for {{INPUT_NUM_FILES}}
* Since {{INPUT_NUM_FILES}} will only be set for {{FileInputFormat}}, putting
the constant there- instead of {{JobContext}}- is probably correct.
FileInputFormat should save #input-files in JobConf
---
Key: MAPREDUCE-1466
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.22.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Priority: Minor
Fix For: 0.22.0
Attachments: MAPREDUCE-1466_yhadoop20-1.patch,
MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20-3.patch,
MAPREDUCE-1466_yhadoop20.patch, mr-1466-trunk-v1.patch
We already track the amount of data consumed by MR applications
(MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the
client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy
to stick in the JobConf during job-submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Attachment: patch-1415-1.txt Patch updated to trunk. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Status: Patch Available (was: Open) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1493) Authorization for job-history pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841807#action_12841807 ] Hadoop QA commented on MAPREDUCE-1493: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437853/MAPREDUCE-1493-20100304.txt against trunk revision 919277. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/console This message is automatically generated. Authorization for job-history pages --- Key: MAPREDUCE-1493 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1493 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobtracker, security Reporter: Vinod K V Assignee: Vinod K V Fix For: 0.22.0 Attachments: MAPREDUCE-1493-20100222.1.txt, MAPREDUCE-1493-20100225.2.txt, MAPREDUCE-1493-20100226.1.txt, MAPREDUCE-1493-20100227.2-ydist.txt, MAPREDUCE-1493-20100227.3-ydist.txt, MAPREDUCE-1493-20100301.1.txt, MAPREDUCE-1493-20100304.txt MAPREDUCE-1455 introduces authorization for most of the Map/Reduce jsp pages and servlets, but left history pages. This JIRA will make sure that authorization checks are made while accessing job-history pages also. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Status: Open (was: Patch Available)
* Setting {{fs.default.name}} prior to creating the {{Job}} should make the
mockito part of the test unnecessary, avoiding pushing the ivy changes through
0.21 and 0.20.
* Consider JUnit4 annotations instead of extending {{TestCase}}
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
m1522_20100301.patch, m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Attachment: M1522-1v20.patch
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Attachment: M1522-1v21.patch
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Attachment: M1522-1.patch
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Status: Patch Available (was: Open)
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1550) UGI.doAs should not be used for getting the history file of jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V resolved MAPREDUCE-1550. -- Resolution: Invalid This bug is there not on trunk but on the ydist branch for which patch was attached at MAPREDUCE-1455. Closing this as invalid. UGI.doAs should not be used for getting the history file of jobs Key: MAPREDUCE-1550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1550 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.22.0 Reporter: Devaraj Das Fix For: 0.22.0 Attachments: 1550-1.patch, 1550-2.1.patch, 1550-2.patch When the jobtracker tries to open a job history file it does a doAs to get the filesystem for the user (that had submitted the job). This should not be done since the job history files are owned by the jobtracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841839#action_12841839 ] Hadoop QA commented on MAPREDUCE-890: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console This message is automatically generated. After HADOOP-4491, the user who started mapred system is not able to run job. - Key: MAPREDUCE-890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Karam Singh Assignee: Ravi Gummadi Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache Directories are created as -: [dr-xrws--- mapred hadoop ] job_200908190916_0002 mapred is not wrtie under this. Even manually I failed to touch file. mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841846#action_12841846
]
Hadoop QA commented on MAPREDUCE-1522:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437996/M1522-1.patch
against trunk revision 919335.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/console
This message is automatically generated.
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1533) reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1533: -- Attachment: mapreduce-1533-v1.4.patch Attaching a patch for review. Instead of doing an in-line String.format(), a wrapper object is passed with all the necessary values to construct the scheduling-info string. The scheduling info is created lazily when toString() method is invoked on the wrapper object. reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects - Key: MAPREDUCE-1533 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1533 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: Rajesh Balamohan Assignee: Amar Kamat Attachments: mapreduce-1533-v1.4.patch When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT executes heartBeat() method heavily. This internally makes a call to CapacityTaskScheduler.updateQSIObjects(). CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() for setting the job scheduling information. Based on the datastructure size of jobQueuesManager and queueInfoMap, the number of times String.format() gets executed becomes very high. String.format() internally does pattern matching which turns to be out very heavy (This was revealed while profiling JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of which String.format() took 46%. Would it be possible to do String.format() only at the time of invoking JobInProgress.getSchedulingInfo?. This might reduce the pressure on JT while processing heartbeats. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841879#action_12841879 ] Hadoop QA commented on MAPREDUCE-1415: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437990/patch-1415-1.txt against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/console This message is automatically generated. With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570 -- Key: MAPREDUCE-1415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, security Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1415-1.txt, patch-1415.txt After MAPREDUCE-856, all localized files are expected to have **0 permissions for the sake of security. This was found by Karam while testing LinuxTaskController functionality after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1564) Document the framework counters
Document the framework counters --- Key: MAPREDUCE-1564 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1564 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Arun C Murthy Fix For: 0.22.0 We need to document the framework counters in the mapred-tutorial. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1565) Need human-readable counter groups for framework job-counters and task-counters
Need human-readable counter groups for framework job-counters and task-counters --- Key: MAPREDUCE-1565 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1565 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker, task Reporter: Arun C Murthy Fix For: 0.22.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1565) Need human-readable counter groups for framework job-counters and task-counters
[ https://issues.apache.org/jira/browse/MAPREDUCE-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1565: - Description: Currently the 'group' is just the class-names of the counters, which is very unfortunate - we'll probably need to have both for a while to maintain compatibility, sigh. Need human-readable counter groups for framework job-counters and task-counters --- Key: MAPREDUCE-1565 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1565 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker, task Reporter: Arun C Murthy Fix For: 0.22.0 Currently the 'group' is just the class-names of the counters, which is very unfortunate - we'll probably need to have both for a while to maintain compatibility, sigh. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12841957#action_12841957
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1522:
---
+1 The changes look good.
Thanks, Chris.
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842060#action_12842060
]
Chris Douglas commented on MAPREDUCE-1522:
--
All test failures are due to Hudson's recent abject uselessness:
{noformat}
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
{noformat}
The former test results hold, as:
# The original run failed only {{TestMRLocalFS}} due to MAPREDUCE-1520
# The only changes are in the unit test, which passes
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I committed this. Thanks, Nicholas!
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1428) Make block size and the size of archive created files configurable.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-1428: -- Attachment: BinaryFileGenerator.java BinaryFileGenerator.java: added a verifier. Make block size and the size of archive created files configurable. --- Key: MAPREDUCE-1428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1428 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.22.0 Attachments: BinaryFileGenerator.java, BinaryFileGenerator.java, BinaryFileGenerator.java Currently the block size used by archives is the default block size of the hdfs filesystem. We need to make it configurable so that the block size can be higher for the part files that archives create. Also, we need to make the size of part files in archives configurable again to make it bigger in size and create less number of such files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842076#action_12842076
]
Hudson commented on MAPREDUCE-1522:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #265 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/265/])
. FileInputFormat may use the default FileSystem for the
input path. Contributed by Tsz Wo (Nicholas), SZE
FileInputFormat may change the file system of an input path
---
Key: MAPREDUCE-1522
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.20.3
Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
m1552_20100223_0.20.patch
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
Path path) uses the default FileSystem but not the FileSystem specified in
the path.
{code}
//org.apache.hadoop.mapreduce.lib.input.FileInputFormat
public static void addInputPath(Job job,
Path path) throws IOException {
Configuration conf = job.getConfiguration();
FileSystem fs = FileSystem.get(conf);
path = path.makeQualified(fs); // the original FileSystem is lost.
...
}
{code}
There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842078#action_12842078
]
Chris Douglas commented on MAPREDUCE-1466:
--
Talked with Luke. Since {{INPUT_NUM_FILES}} is shared between both
implementations of {{FileInputFormat}}, most config keys were moved to
JobContext in MAPREDUCE-849 anyway, and config keys are undocumented in our
current regime: the current patch is correct.
+1
FileInputFormat should save #input-files in JobConf
---
Key: MAPREDUCE-1466
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.22.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Priority: Minor
Fix For: 0.22.0
Attachments: MAPREDUCE-1466_yhadoop20-1.patch,
MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20-3.patch,
MAPREDUCE-1466_yhadoop20.patch, mr-1466-trunk-v1.patch
We already track the amount of data consumed by MR applications
(MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the
client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy
to stick in the JobConf during job-submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1558) specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842090#action_12842090 ] Owen O'Malley commented on MAPREDUCE-1558: -- There is no need to create the JobConf. Just do the get out of the conf. specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612) - Key: MAPREDUCE-1558 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1558 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: MAPREDUCE-1558.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1566) Need to add a mechanism to import tokens and secrets into a submitted job.
Need to add a mechanism to import tokens and secrets into a submitted job. -- Key: MAPREDUCE-1566 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1566 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.22.0 We need to include tokens and secrets into a submitted job. I propose adding a configuration attribute that when pointed at a token storage file will include the tokens and secrets from that token storage file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1567) Sharing Credentials between JobConfs leads to unintentional sharing of credentials
Sharing Credentials between JobConfs leads to unintentional sharing of credentials -- Key: MAPREDUCE-1567 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1567 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.22.0 Currently, if code does new JobConf(jobConf), it will share the Credentials. That leads to unintentional sharing. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1568) TrackerDistributedCacheManager should do deleteLocalPath asynchronously
TrackerDistributedCacheManager should do deleteLocalPath asynchronously --- Key: MAPREDUCE-1568 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1568 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 TrackerDistributedCacheManager.deleteCache() has been improved: MAPREDUCE-1302 makes TrackerDistributedCacheManager rename the caches in the main thread and then delete them in the background MAPREDUCE-1098 avoids global locking while do the renaming (renaming lots of directories can also takes a long time) But the deleteLocalCache is still in the main thread of TaskRunner.run(). So it will still slow down the task which triggers the deletion (originally this will blocks all tasks, but it is fixed by MAPREDUCE-1098). Other tasks do not wait for the deletion. The task which triggers the deletion should not wait for this either. TrackerDistributedCacheManager should do deleteLocalPath() asynchronously. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-653) distcp can support bandwidth limiting
[ https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12842218#action_12842218 ] eric baldeschwieler commented on MAPREDUCE-653: --- So how will this behave if distcp gets 100 slots and is limited to 100mb? Will you use 100 slots badly? Typically we've handled the need for throttling by controlling the bandwidth / client and the number of clients. This lets you reason about the utilization of the clients bandwidth. This would imply a fixed number of maps that pull files to copy off a queue maintained by the distcp client. You'd need to handle recovery carefully, but that could probably be done at the distcp client (the slave signals done, the client does a final move and updates the queue). distcp can support bandwidth limiting - Key: MAPREDUCE-653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-653 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distcp Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch distcp should support an option for user to specify the bandwidth limit for the distcp job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
