[jira] Commented: (MAPREDUCE-372) Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790030#action_12790030 ] Amareshwari Sriramadasu commented on MAPREDUCE-372: --- All core and contrib tests passed on my machine Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api. --- Key: MAPREDUCE-372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-372 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: mapred-372.patch, mapred-372.patch, mapred-372.patch, patch-372-1.txt, patch-372-2.txt, patch-372-3.txt, patch-372-4.txt, patch-372.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1274) The completed job web ui urls include full path names to the local file system on the JobTracker.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790032#action_12790032 ] Amareshwari Sriramadasu commented on MAPREDUCE-1274: bq. the web ui for MapReduce in 0.21.0-dev include a path to a local file in the url: The web ui url for retired jobs and history url have this. bq. I suspect the answer is applying MAPREDUCE-1185 back to 0.21. MAPREDUCE-1185 does not solve the above mentioned problem. It only redirects the job url to history url. I think one solution is MAPREDUCE-323, where job history file depends only on jobid. The completed job web ui urls include full path names to the local file system on the JobTracker. - Key: MAPREDUCE-1274 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1274 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.21.0 Reporter: Owen O'Malley Priority: Blocker Fix For: 0.21.0 Currently, the web ui for MapReduce in 0.21.0-dev include a path to a local file in the url: http://jt.foo.com:50030/jobdetailshistory.jsp?jobid=job_200912012129_0001logFile=file%3A%2Fopt%2Flocal%2Fowen%2Fhadoop%2Frun%2Flogs%2Fhistory%2Fdone%2Fjob_200912012129_0001_oom This implies a security bug where the user uses logFile=/etc/passwd or some other annoying trick. I suspect the answer is applying MAPREDUCE-1185 back to 0.21. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-372) Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790039#action_12790039 ] Hadoop QA commented on MAPREDUCE-372: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427693/patch-372-4.txt against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/console This message is automatically generated. Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api. --- Key: MAPREDUCE-372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-372 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: mapred-372.patch, mapred-372.patch, mapred-372.patch, patch-372-1.txt, patch-372-2.txt, patch-372-3.txt, patch-372-4.txt, patch-372.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-1143: - Attachment: MAPRED-1143-ydist-9.patch incorporated amareshwari's comment runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh Priority: Blocker Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-1143: - Attachment: MAPRED-1143-7.patch runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh Priority: Blocker Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-372) Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790044#action_12790044 ] Amareshwari Sriramadasu commented on MAPREDUCE-372: --- test failures are due to MAPREDUCE-1275. All the core and contrib tests passed on my machine. Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api. --- Key: MAPREDUCE-372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-372 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: mapred-372.patch, mapred-372.patch, mapred-372.patch, patch-372-1.txt, patch-372-2.txt, patch-372-3.txt, patch-372-4.txt, patch-372.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-1143: - Status: Patch Available (was: Open) runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh Priority: Blocker Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1291) JobTracker fails to remove setup tip mapping from taskidToTIPMap if the job gets killed before the setup returns
JobTracker fails to remove setup tip mapping from taskidToTIPMap if the job gets killed before the setup returns Key: MAPREDUCE-1291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1291 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Amar Kamat Priority: Critical Here is the scenario : 1) job inits 2) setup task is launched on tt1 and an entry is made in taskidToTIPMap 3) job is killed 4) cleanup gets launched on tt2 5) cleanup returns KILLING the job and removing all the *completed* setup/map/reduce task mappings from taskidToTIPMap. Here the setup is still RUNNING state. 6) job retires and all the map/reduce mappings from taskidToTIPMap are removed In the end the setup tip still lingers in the taskidToTIPMap map. Because of the backreference from the tip to jip, the whole job stays in memory forever. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Attachment: mapreduce-698-trunk-4.patch Added some warnings if it's configured with max min, fixed layout in servlet to match current version. Also, I believe the tests have been fixed in trunk, so this should pass release audit now. Per-pool task limits for the fair scheduler --- Key: MAPREDUCE-698 URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Kevin Peterson Fix For: 0.21.0 Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, mapreduce-698-trunk.patch The fair scheduler could use a way to cap the share of a given pool similar to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Status: Open (was: Patch Available) cancel and resubmit to pick up newer patch. Per-pool task limits for the fair scheduler --- Key: MAPREDUCE-698 URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Kevin Peterson Fix For: 0.21.0 Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, mapreduce-698-trunk.patch The fair scheduler could use a way to cap the share of a given pool similar to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Status: Patch Available (was: Open) Should be ready to go. Per-pool task limits for the fair scheduler --- Key: MAPREDUCE-698 URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Kevin Peterson Fix For: 0.21.0 Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, mapreduce-698-trunk.patch The fair scheduler could use a way to cap the share of a given pool similar to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790065#action_12790065 ] Hadoop QA commented on MAPREDUCE-1143: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427890/MAPRED-1143-6.patch against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/console This message is automatically generated. runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh Priority: Blocker Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1143: - Affects Version/s: 0.20.1 Fix Version/s: 0.21.0 runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-1143: Assignee: rahul k singh runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790078#action_12790078 ] Arun C Murthy commented on MAPREDUCE-1143: -- +1 for the patch. runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790095#action_12790095 ] Hemanth Yamijala commented on MAPREDUCE-1186: - Amarsri, Vinod and I discussed the trunk patch a bit. The current implementation attempts to work as follows: - Before task launch, the task controller is launched to secure localized cache files. Previously, all files under $mapred-local-dir/$user/taskTracker/archive were secured. Obviously, we are trying to fix that in the context of this JIRA. - The patch lists the directories under $mapred-local-dir/$user/taskTracker/archive, (which after MAPREDUCE-1098, is the list of random id directories that were localized). - For each directory, if the path is not already secured, it secures it recursively. This approach has a race condition that we identified: - Say a task has localized a file and has launched the task controller to secure the path, and the task controller is currently under operation. - In parallel, say another task localized another file into a different random id directory. - The task controller could get the random id directory created by the second task when it is listing directories and set permissions for it. However, this directory does not contain fully localized files and hence it would be incompletely localized. The key problem here is that this approach does not have a real idea of what files were localized by a task as part of the distributed cache. One way to fix that would be to pass the paths to the task controller, as a list of random id directories under $mapred-local-dir/$user/taskTracker/archive that were localized in this task. This is what I suggested in the proposal above. However, there are a few problems with this proposal as well: - How do we get the list of these paths ? There's currently no way exposed by distributed cache about these files. - This could be a huge list, if several tens of files are being localized in a task. How would we transfer all this info to the task-controller ? A huge command line of paths to the task controller could be unmanageable, hit some command line length limits, etc. Other approaches (like transferring the info through a file) would also be cumbersome. - It could result in duplicate work. Say if two tasks running in parallel are sharing a file, both of them would get the random id directory to secure, and both would try and secure the path. To solve these problems, I am proposing the following: - Change the directory structure for localized cache files as: $mapred-local-dir/$user/taskTracker/archive/$task-id, where task-id is for the task attempt on behalf of which localization is happening. Note that a task could use localized files that have already been localized for another task-id. Since a cache entry stores the full path for a cache key, it can retrieve this information. - Move securing the cache file path in the same code path as where localization of the cache files happens. The last point is actually important in this new proposal, because without that, we might have a situation that a task could use files that have been localized by a prior task-id, but is not yet secured. And if we don't wait for that, we would have incompletely secured cache files in use. One drawback I can think of this approach is that the new task-id directory in the path might give a wrong impression that the files localized under it are all the files used by the task in distributed cache. But clearly, files localized under other task-ids could be used as well. Comments on this proposal ? While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-1.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-962) NPE in ProcfsBasedProcessTree.destroy()
[
https://issues.apache.org/jira/browse/MAPREDUCE-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-962:
Affects Version/s: 0.21.0
NPE in ProcfsBasedProcessTree.destroy()
---
Key: MAPREDUCE-962
URL: https://issues.apache.org/jira/browse/MAPREDUCE-962
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.21.0
Reporter: Vinod K V
Assignee: Ravi Gummadi
Fix For: 0.21.0
Attachments: HADOOP-6232.patch, MR-962.patch, MR-962.v1.1.patch,
MR-962.v1.patch
This causes the following exception in TaskMemoryManagerThread. I observed
this while running TestTaskTrackerMemoryManager.
{code}
2009-09-02 12:08:25,835 WARN mapred.TaskMemoryManagerThread
(TaskMemoryManagerThread.java:run(239)) - \
Uncaught exception in TaskMemoryManager while managing memory of
attempt_20090902120812252_0001_m_03_0 : \
java.lang.NullPointerException
at
org.apache.hadoop.util.ProcfsBasedProcessTree.assertPidPgrpidForMatch(ProcfsBasedProcessTree.java:234)
at
org.apache.hadoop.util.ProcfsBasedProcessTree.assertAndDestroyProcessGroup(ProcfsBasedProcessTree.java:257)
at
org.apache.hadoop.util.ProcfsBasedProcessTree.destroy(ProcfsBasedProcessTree.java:286)
at
org.apache.hadoop.mapred.TaskMemoryManagerThread.run(TaskMemoryManagerThread.java:229)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790108#action_12790108 ] rahul k singh commented on MAPREDUCE-1143: -- most of the test cases are failing in the build. with org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/ipc/VersionedProtocol; java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/ipc/VersionedProtocol; This is happening on the trunk too , i.e , without applying the patch . This is in no way related to the current fix. runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1292) many testcases are failing in trunk with org.apache.hadoop.ipc.RPC.waitForProxy error.
many testcases are failing in trunk with org.apache.hadoop.ipc.RPC.waitForProxy error. -- Key: MAPREDUCE-1292 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1292 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1292) many testcases are failing in trunk with org.apache.hadoop.ipc.RPC.waitForProxy error.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790110#action_12790110 ] rahul k singh commented on MAPREDUCE-1292: -- many testcases are failing due to org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/ipc/VersionedProtocol; java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/ipc/VersionedProtocol; exception. many testcases are failing in trunk with org.apache.hadoop.ipc.RPC.waitForProxy error. -- Key: MAPREDUCE-1292 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1292 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790111#action_12790111 ] rahul k singh commented on MAPREDUCE-1143: -- opened jira [1292|https://issues.apache.org/jira/browse/MAPREDUCE-1292] for the above issue runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790112#action_12790112 ] Ravi Gummadi commented on MAPREDUCE-1252: - In HADOOP-6105, old key takes precedence irrespective of the order in which resources are loaded. I propose that we resolve based on the order of loading of resources irrespective of old/new key is seen. Thoughts ? Shuffle deadlocks on wrong number of maps - Key: MAPREDUCE-1252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.21.0, 0.22.0 Reporter: Owen O'Malley Assignee: Owen O'Malley Priority: Blocker Fix For: 0.21.0, 0.22.0 Attachments: mr-1252.patch The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing: 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790119#action_12790119
]
Ankit Modi commented on MAPREDUCE-1214:
---
Hi Jeff
I am trying to access counter values in code in pig. This is how it is done for
Hadoop MR mode.
I am not able to replicate the process in Hadoop Local mode as we don't have an
AssignedJobId.
{code}
RunningJob rj = jobClient.getJob( job.getAssignedJobID() );
Counters counters = rj.getCounters();
Counters.Group taskgroup =
counters.getGroup(org.apache.hadoop.mapred.Task$Counter);
value = taskgroup.getCounterForName(REDUCE_OUTPUT_RECORDS).getCounter();
value2 = taskgroup.getCounterForName(MAP_OUTPUT_RECORDS).getCounter();
{code}
Is this the correct way, or am I missing something ?
Add support for counters in Hadoop Local Mode
-
Key: MAPREDUCE-1214
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Ankit Modi
Currently there is no support for counters ( Records and Bytes written ) in
Hadoop Local Mode.
Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790122#action_12790122 ] Hadoop QA commented on MAPREDUCE-1143: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427899/MAPRED-1143-7.patch against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/console This message is automatically generated. runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790123#action_12790123 ] Hadoop QA commented on MAPREDUCE-698: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427904/mapreduce-698-trunk-4.patch against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/console This message is automatically generated. Per-pool task limits for the fair scheduler --- Key: MAPREDUCE-698 URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Kevin Peterson Fix For: 0.21.0 Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, mapreduce-698-trunk.patch The fair scheduler could use a way to cap the share of a given pool similar to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790165#action_12790165 ] rahul k singh commented on MAPREDUCE-1143: -- tested the fix for yhadoop , manually, it works fine. runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790166#action_12790166 ] Jeff Zhang commented on MAPREDUCE-1214: --- Hi Ankit, I checked the code in pig, and find that you use the JobClient the HExecutionEngine, but actually the JobClient in org.apache.hadoop.mapred.jobcontrol.Job submit the job. And each JobClient will create a LocalJobRunner, so that means you use the wrong JobClient/LocalJobRunner to get RunningJob. This won't affect the mapreduce mode, because you only have one master node as the JobTracker. Forgive me to attach the Pig patch here. Ankit, do you have the JIRA for this issue in Pig, I can move the patch there. Add support for counters in Hadoop Local Mode - Key: MAPREDUCE-1214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ankit Modi Attachments: MAPREDUCE-1214.patch Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode. Pig requires to provide counters to user when running in Hadoop Local Mode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1214) Add support for counters in Hadoop Local Mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated MAPREDUCE-1214: -- Attachment: MAPREDUCE-1214.patch Add support for counters in Hadoop Local Mode - Key: MAPREDUCE-1214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ankit Modi Attachments: MAPREDUCE-1214.patch Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode. Pig requires to provide counters to user when running in Hadoop Local Mode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1259) Add SureLogic annotations' jar into Ivy and Eclipse configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwin Chan updated MAPREDUCE-1259: -- Attachment: mapReduce_3.1.0.patch This patches the Ivy and Eclipse configs to download and use the promises-3.1.0.jar from Maven Add SureLogic annotations' jar into Ivy and Eclipse configs --- Key: MAPREDUCE-1259 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1259 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Attachments: mapReduce_3.1.0.patch In order to use SureLogic analysis tools and allow their concurrency analysis annotations in HDFS code the annotations library has to be automatically pulled from a Maven repo. Also, it has to be added to Eclipse .classpath template. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1207) Allow admins to set java options for map/reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790228#action_12790228
]
Steve Loughran commented on MAPREDUCE-1207:
---
Given IPv6 is so much trouble to Hadoop, maybe that particular setting should
be everywhere anyway
Allow admins to set java options for map/reduce tasks
-
Key: MAPREDUCE-1207
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1207
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Arun C Murthy
It will be useful for allow cluster-admins to set some java options for child
map/reduce tasks.
E.g. We've had to ask users to set -Djava.net.preferIPv4Stack=true in their
jobs, it would be nice to do it for all users in such scenarios even when
people override mapred.child.{map|reduce}.java.opts but forget to add this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1292) many testcases are failing in trunk with org.apache.hadoop.ipc.RPC.waitForProxy error.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790253#action_12790253 ] Doug Cutting commented on MAPREDUCE-1292: - Have you recompiled, i.e., 'ant clean'? HADOOP-6422 reqires recompilation of sources. many testcases are failing in trunk with org.apache.hadoop.ipc.RPC.waitForProxy error. -- Key: MAPREDUCE-1292 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1292 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: rahul k singh -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1247) Send out-of-band heartbeat to avoid fake lost tasktracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao reassigned MAPREDUCE-1247: - Assignee: ZhuGuanyin Send out-of-band heartbeat to avoid fake lost tasktracker - Key: MAPREDUCE-1247 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1247 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: ZhuGuanyin Assignee: ZhuGuanyin Currently the TaskTracker report task status to jobtracker through heartbeat, sometimes if the tasktracker lock the tasktracker to do some cleanup job, like remove task temp data on disk, the heartbeat thread would hang for a long time while waiting for the lock, so the jobtracker just thought it had lost and would reschedule all its finished maps or un finished reduce on other tasktrackers, we call it fake lost tasktracker, some times it doesn't acceptable especially when we run some large jobs. So We introduce a out-of-band heartbeat mechanism to send an out-of-band heartbeat in that case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1084) Implementing aspects development and fault injeciton framework for MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1084: -- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've committed the latest patch with my fixes. Thanks Sreekanth! Implementing aspects development and fault injeciton framework for MapReduce Key: MAPREDUCE-1084 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1084 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, test Reporter: Konstantin Boudnik Assignee: Sreekanth Ramakrishnan Fix For: 0.22.0 Attachments: mapreduce-1084-1-withoutsvnexternals.patch, mapreduce-1084-1.patch, mapreduce-1084-2.patch, mapreduce-1084-3.patch, mapreduce-1084-5.patch, mapreduce-1084-6-withRandom.patch, mapreduce-1084-6.patch, mapreduce-1084-final.patch Similar to HDFS-435 and HADOOP-6204 this JIRA will track the introduction of injection framework for MapReduce. After HADOOP-6204 is in place this particular modification should be very trivial and would take importing (via svn:external) of src/test/build and some tweaking of the build.xml file -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1293) AutoInputFormat doesn't work with non-default FileSystems
[ https://issues.apache.org/jira/browse/MAPREDUCE-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Hitchcock updated MAPREDUCE-1293: Status: Patch Available (was: Open) AutoInputFormat doesn't work with non-default FileSystems - Key: MAPREDUCE-1293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1293 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Reporter: Andrew Hitchcock Attachments: MAPREDUCE-1293.txt AutoInputFormat uses the wrong FileSystem.get() method when getting a reference to a FileSystem object. AutoInputFormat gets the default FileSystem, so this method breaks if the InputSplit's path is pointing to a different FileSystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1293) AutoInputFormat doesn't work with non-default FileSystems
[ https://issues.apache.org/jira/browse/MAPREDUCE-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Hitchcock updated MAPREDUCE-1293: Attachment: MAPREDUCE-1293.txt Switching to the proper .get() method fixes the issue. AutoInputFormat doesn't work with non-default FileSystems - Key: MAPREDUCE-1293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1293 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Reporter: Andrew Hitchcock Attachments: MAPREDUCE-1293.txt AutoInputFormat uses the wrong FileSystem.get() method when getting a reference to a FileSystem object. AutoInputFormat gets the default FileSystem, so this method breaks if the InputSplit's path is pointing to a different FileSystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof can cause JobClient to report job failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-1131:
-
Attachment: MAPREDUCE-1131.6.patch
New patch with tests updated per Vinod's request.
Ran test-patch locally:
{code}
[exec] +1 overall.- -
[exec]-
[exec] +1 @author. The patch does not contain any @author tags.
[exec]-
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]-
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]-
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]-
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]-
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]-
{code}
Using profilers other than hprof can cause JobClient to report job failure
--
Key: MAPREDUCE-1131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch,
MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch,
MAPREDUCE-1131.patch
If task profiling is enabled, the JobClient will download the {{profile.out}}
file created by the tasks under profile. If this causes an IOException, the
job is reported as a failure to the client, even though all the tasks
themselves may complete successfully. The expected result files are assumed
to be generated by hprof. Using the profiling system with other profilers
will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof can cause JobClient to report job failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-1131:
-
Status: Patch Available (was: Open)
Using profilers other than hprof can cause JobClient to report job failure
--
Key: MAPREDUCE-1131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch,
MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch,
MAPREDUCE-1131.patch
If task profiling is enabled, the JobClient will download the {{profile.out}}
file created by the tasks under profile. If this causes an IOException, the
job is reported as a failure to the client, even though all the tasks
themselves may complete successfully. The expected result files are assumed
to be generated by hprof. Using the profiling system with other profilers
will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1124) TestGridmixSubmission fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1124: - Attachment: M1124-y20-1.patch TestGridmixSubmission fails sometimes - Key: MAPREDUCE-1124 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1124 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.21.0 Reporter: Amareshwari Sriramadasu Assignee: Chris Douglas Fix For: 0.21.0 Attachments: M1124-0.patch, M1124-1.patch, M1124-2.patch, M1124-3.patch, M1124-4.patch, M1124-5.patch, M1124-y20-1.patch, M1124-y20.patch, TEST-org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.txt TestGridmixSubmission fails sometimes with following error : Mismatched output bytes 4547848/4561267 junit.framework.AssertionFailedError: Mismatched output bytes 4547848/4561267 at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1294) Build fails to pull latest hadoop-core-* artifacts
[ https://issues.apache.org/jira/browse/MAPREDUCE-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1294: -- Attachment: MAPREDUCE-1294.patch The patch is the same as one committed with HDFS-825. The patch increases ivy-resolve phase for a couple of seconds. But now the builds are consistent with the published artifacts. Build fails to pull latest hadoop-core-* artifacts -- Key: MAPREDUCE-1294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1294 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Critical Attachments: MAPREDUCE-1294.patch This is the same as HDFS-825 for mapreduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1294) Build fails to pull latest hadoop-core-* artifacts
[ https://issues.apache.org/jira/browse/MAPREDUCE-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1294: -- Status: Patch Available (was: Open) Ready for the verification. Build fails to pull latest hadoop-core-* artifacts -- Key: MAPREDUCE-1294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1294 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Critical Attachments: MAPREDUCE-1294.patch This is the same as HDFS-825 for mapreduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Attachment: MAPREDUCE-1213.3.patch This one uses the newly-committed AsyncDiskService from common. TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch, MAPREDUCE-1213.3.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Status: Open (was: Patch Available) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch, MAPREDUCE-1213.3.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Attachment: MAPREDUCE-1213.3.patch This one uses the AsyncDiskService from common. TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch, MAPREDUCE-1213.3.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Attachment: (was: MAPREDUCE-1213.3.patch) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch, MAPREDUCE-1213.3.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1295) We need a job trace manipulator to build gridmix runs.
We need a job trace manipulator to build gridmix runs. -- Key: MAPREDUCE-1295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Dick King Rumen produces job traces, which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ]. It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a folder, a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Attachment: (was: MAPREDUCE-1213.3.patch) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Attachment: (was: MAPREDUCE-1213.3.patch) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1213: -- Attachment: MAPREDUCE-1213.3.patch TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch, MAPREDUCE-1213.3.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to build gridmix runs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dick King updated MAPREDUCE-1295: - Attachment: mapreduce-1297--2009-12-14.patch We need a job trace manipulator to build gridmix runs. -- Key: MAPREDUCE-1295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Dick King Assignee: Dick King Attachments: mapreduce-1297--2009-12-14.patch Rumen produces job traces, which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ]. It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a folder, a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof can cause JobClient to report job failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790531#action_12790531
]
Hadoop QA commented on MAPREDUCE-1131:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427970/MAPREDUCE-1131.6.patch
against trunk revision 890531.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/console
This message is automatically generated.
Using profilers other than hprof can cause JobClient to report job failure
--
Key: MAPREDUCE-1131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch,
MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch,
MAPREDUCE-1131.patch
If task profiling is enabled, the JobClient will download the {{profile.out}}
file created by the tasks under profile. If this causes an IOException, the
job is reported as a failure to the client, even though all the tasks
themselves may complete successfully. The expected result files are assumed
to be generated by hprof. Using the profiling system with other profilers
will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1259) Add SureLogic annotations' jar into Ivy and Eclipse configs
[
https://issues.apache.org/jira/browse/MAPREDUCE-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790553#action_12790553
]
Konstantin Boudnik commented on MAPREDUCE-1259:
---
Edwin, why's there this modification in
{{src/contrib/index/ivy/libraries.properties}} ?
Add SureLogic annotations' jar into Ivy and Eclipse configs
---
Key: MAPREDUCE-1259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
Attachments: mapReduce_3.1.0.patch
In order to use SureLogic analysis tools and allow their concurrency analysis
annotations in HDFS code the annotations library has to be automatically
pulled from a Maven repo. Also, it has to be added to Eclipse .classpath
template.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-64) Map-side sort is hampered by io.sort.record.percent
[
https://issues.apache.org/jira/browse/MAPREDUCE-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790569#action_12790569
]
Hong Tang commented on MAPREDUCE-64:
The design is quite clever and elegant. I like it. The code is a clean, but a
bit tricky to understand (more on this later with some of my suggestions on
refactory).
- MapOutputBuffer.collect: The logic of calculating the equator seems to be
missing a multipication of METASIZE. Should be:
{code}
final int newPos = (bufindex +
Math.max(2 * METASIZE - 1,
Math.min(distkvi / 2, distkvi / (METASIZE + avgRec) *
METASIZE)))
{code}
- Buffer.write(byte[], int, int): blockwrite = distkvi distkve should be
blockwrite = distkvi = distkve
- A potential inefficiency if we encounter a large record when there are few
(but not zero) records in the buffer - this would lead to these few records
written out as a single spill. A better way is to spill out the single large
record, and continue accumulating records after that. This should be a very
rare corner case so may not need to be addressed in this jira. Would be nice to
mark it with TODO in the comments.
- Any particular reason to shut down the thread in Buffer.flush() rather than
Buffer.close()?
- In SpillThread: if (bufend bufindex bufindex bufstart) should
probably be if (bufend bufstart) {
- In TestMapCollection: uniform random is used to determine how many bytes to
write in serialization, and to determine key/value size for RandomFactory. This
is less desirable in the sense that very small values are not sufficiently
tested. Suggest to change to a distribution that gives more weight to small
values e.g. (min + Math.exp(random.nextDouble()*Math.log(max-min))).
I also have a couple of suggestions on refactoring the code to make it more
readable:
- Separate the sets of variables used by main thread for writing from the set
of variables for the spill threads for spilling. (Currently kvend and bufend
are used in two different context: when there is a spill active or when there
is not).
- Related to the above, adding a variable called spillExists to describe the
state when there is a spill buffer. The life time of spillExists==TRUE covers
that of spillInProgress==TRUE.
- suggest to change the direct (idx+offset) based access to kvmeta to method
calls.
- Suggest to refactor the logic on marking a spill region.
Other very minor nits:
- MapOutputBuffer.collect: it would be nice to spell out the invariance that
there are always METASIZE bytes available beyond kvindex.
- MapOutputBuffer: document the use of bufferRemaining as a hint whether we
*may* need to block and spill. If bufferRemaining=0, there is guaranteed space
for us to continue write.
- BlockBuffer is only usable inside MapOutputBuffer, suggest remove the
constructor BlockBuffer(OutputStream).
- Suggest rename BlockBuffer.reset() to BlockBuffer.shiftKeyBuffer().
- Suggest to add a note to Buffer.write(byte[], int, int) that the checking of
bufferRemaining should not be bypassed even if len==0.
Map-side sort is hampered by io.sort.record.percent
---
Key: MAPREDUCE-64
URL: https://issues.apache.org/jira/browse/MAPREDUCE-64
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun C Murthy
Assignee: Chris Douglas
Attachments: M64-0.patch, M64-0i.png, M64-1.patch, M64-1i.png,
M64-2.patch, M64-2i.png, M64-3.patch, M64-4.patch
Currently io.sort.record.percent is a fairly obscure, per-job configurable,
expert-level parameter which controls how much accounting space is available
for records in the map-side sort buffer (io.sort.mb). Typically values for
io.sort.mb (100) and io.sort.record.percent (0.05) imply that we can store
~350,000 records in the buffer before necessitating a sort/combine/spill.
However for many applications which deal with small records e.g. the
world-famous wordcount and it's family this implies we can only use 5-10% of
io.sort.mb i.e. (5-10M) before we spill inspite of having _much_ more memory
available in the sort-buffer. The word-count for e.g. results in ~12 spills
(given hdfs block size of 64M). The presence of a combiner exacerbates the
problem by piling serialization/deserialization of records too...
Sure, jobs can configure io.sort.record.percent, but it's tedious and
obscure; we really can do better by getting the framework to automagically
pick it by using all available memory (upto io.sort.mb) for either the data
or accounting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790581#action_12790581 ] rahul k singh commented on MAPREDUCE-1143: -- [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] This fix doesnt have testcase , as it requires a significant change , we have provided for the trunk , manual testing is done to make sure that things are fine. for eclipse classpath its a known issue for 20 internal patch. runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-815) Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro Serialization
[
https://issues.apache.org/jira/browse/MAPREDUCE-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790583#action_12790583
]
Jacob Rideout commented on MAPREDUCE-815:
-
What is the current line of thought on how keys and values will interact with
the schema for an avro file? Is the intention that there would be a master
schema that encapsulated the key/values similar to:
{code}
{ type : record,
fields : [
{ name : KEY, type : record },
{ name : VALUE, type : record }
]}
{code}
What about files created without this master schema; would the key return a
null object? Byte offset in a schema of type long ?
Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro
Serialization
--
Key: MAPREDUCE-815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-815
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
MapReduce needs AvroInputFormat similar to other InputFormats like
TextInputFormat to be able to use avro serialization in hadoop. Similarly
AvroOutputFormat is needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1294) Build fails to pull latest hadoop-core-* artifacts
[ https://issues.apache.org/jira/browse/MAPREDUCE-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790584#action_12790584 ] Hadoop QA commented on MAPREDUCE-1294: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427975/MAPREDUCE-1294.patch against trunk revision 890593. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/193/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/193/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/193/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/193/console This message is automatically generated. Build fails to pull latest hadoop-core-* artifacts -- Key: MAPREDUCE-1294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1294 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Critical Attachments: MAPREDUCE-1294.patch This is the same as HDFS-825 for mapreduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1277) Streaming job should support other characterset in user's stderr log, not only utf8
[ https://issues.apache.org/jira/browse/MAPREDUCE-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhuGuanyin updated MAPREDUCE-1277: -- Attachment: streaming-1277-new.patch regenerate the patch using svn diff at root dir, thanks. Streaming job should support other characterset in user's stderr log, not only utf8 --- Key: MAPREDUCE-1277 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1277 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.21.0 Reporter: ZhuGuanyin Assignee: ZhuGuanyin Fix For: 0.21.0 Attachments: streaming-1277-new.patch, streaming-1277.patch Current implementation in streaming only support utf8 encoded user stderr log, it should encode free to support other characterset. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[
https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790589#action_12790589
]
Vinod K V commented on MAPREDUCE-1213:
--
The class {{org.apache.hadoop.mapred.CleanupQueue}} is very similar to
{{MRAsyncDiskService}}. I think we should merge their functionality here or in
another issue.
TaskTrackers restart is very slow because it deletes distributed cache
directory synchronously
--
Key: MAPREDUCE-1213
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.1
Reporter: dhruba borthakur
Assignee: Zheng Shao
Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch,
MAPREDUCE-1213.3.patch
We are seeing that when we restart a tasktracker, it tries to recursively
delete all the file in the distributed cache. It invoked
FileUtil.fullyDelete() which is very very slow. This means that the
TaskTracker cannot join the cluster for an extended period of time (upto 2
hours for us). The problem is acute if the number of files in a distributed
cache is a few-thousands.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1165) SerialUtils.hh: __PRETTY_FUNCTION__ is a GNU extension and not portable
[ https://issues.apache.org/jira/browse/MAPREDUCE-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790607#action_12790607 ] Eli Collins commented on MAPREDUCE-1165: +1 Patch looks good. Using this C99 ism is should be fine on linux, solaris and osx. I tested the patch on linux after applying MAPREDUCE-1251. SerialUtils.hh: __PRETTY_FUNCTION__ is a GNU extension and not portable --- Key: MAPREDUCE-1165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Attachments: MAPREREDUCE-1165.patch SerialUtils.hh uses __PRETTY_FUNCTION__ to print the name of the function during an assertion. That is a GNU extension and is not portable across compilers. [C99 defines __func__, which should probably be used instead.] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
