[jira] [Updated] (MAPREDUCE-2127) mapreduce trunk builds are filing on hudson ..
[ https://issues.apache.org/jira/browse/MAPREDUCE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-2127: -- Affects Version/s: 0.23.0 0.20.204.0 0.20.203.1 Status: Patch Available (was: Open) Here is a reworked patch to make it compile on fedora 15. It was tested and working on Centos 5, fedora 15 and ubuntu maverick 10.10. mapreduce trunk builds are filing on hudson .. --- Key: MAPREDUCE-2127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2127 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 0.20.203.1, 0.20.204.0, 0.23.0 Reporter: Giridharan Kesavan Attachments: MAPREDUCE-2127.patch https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lssl... no [exec] configure: error: Cannot find libssl.so [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2127) mapreduce trunk builds are filing on hudson ..
[ https://issues.apache.org/jira/browse/MAPREDUCE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated MAPREDUCE-2127: -- Attachment: MAPREDUCE-2127.patch HMAC_Init seems to be defined in libcrypto and fedora policy regarding DSO linking prevent it from being picked up as explainied a few comments above. mapreduce trunk builds are filing on hudson .. --- Key: MAPREDUCE-2127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2127 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 0.20.203.1, 0.20.204.0, 0.23.0 Reporter: Giridharan Kesavan Attachments: MAPREDUCE-2127.patch, MAPREDUCE-2127.patch https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lssl... no [exec] configure: error: Cannot find libssl.so [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2555) JvmInvalidate errors in the gridmix TT logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2555: - Fix Version/s: (was: 0.20.205.0) 0.20.204.0 JvmInvalidate errors in the gridmix TT logs --- Key: MAPREDUCE-2555 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2555 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Fix For: 0.20.204.0 Attachments: MAPREDUCE-2555-20security-v2.patch, MAPREDUCE-2555-20security.patch Observing a lot of jvmValidate exceptions in TT logs for grid mix run 2011-04-28 02:00:37,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 46121, call statusUpdate(attempt_201104270735_5993_m_003305_0, org.apache.hadoop.mapred.MapTaskStatus@1840a9c, org.apache.hadoop.mapred.JvmContext@1d4ab6b) from 127.0.0.1:50864: error: java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201104270735_5993_m_003305_0, with JvmId: jvm_201104270735_5993_m_103399012gsbl20430: java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201104270735_5993_m_003305_0, with JvmId: jvm_201104270735_5993_m_103399012gsbl20430: -- at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1386) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1384) * -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated MAPREDUCE-2550: - Attachment: MAPREDUCE-2550-2.patch Same patch with HADOOP_PREFIX removed from mapred-config.sh. bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550-2.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2524: - Fix Version/s: 0.20.204.0 Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle -- Key: MAPREDUCE-2524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Fix For: 0.20.204.0, 0.20.205.0 Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2514) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2514: - Fix Version/s: 0.20.204.0 ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log Key: MAPREDUCE-2514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2514 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.204.0, 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2514-branch-0.20-security.patch, MAPREDUCE-2514-trunk.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2495) The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2495: - Fix Version/s: 0.20.204.0 The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason - Key: MAPREDUCE-2495 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2495 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Affects Versions: 0.21.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Minor Fix For: 0.20.204.0, 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2495-20.20X-V1.patch, MAPREDUCE-2495-20.20X-V2.patch, MAPREDUCE-2495-20.20X-V3.patch, MAPREDUCE-2495-20.20X-V4.patch, MAPREDUCE-2495-v1.patch, MAPREDUCE-2495-v2.patch, MAPREDUCE-2495-v3.patch, MAPREDUCE-2495-v4.patch The cleanup thread in the distributed cache handles IOExceptions and the like correctly, but just to be a bit more defensive it would be good to monitor the thread, and check that it is still alive regularly, so that the distributed cache does not fill up the entire disk on the node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2529) Recognize Jetty bug 1342 and handle it
[ https://issues.apache.org/jira/browse/MAPREDUCE-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2529: - Fix Version/s: 0.20.204.0 Recognize Jetty bug 1342 and handle it -- Key: MAPREDUCE-2529 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2529 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.204.0, 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.20.204.0, 0.20.205.0, 0.23.0 Attachments: M2529-1-20s.patch, M2529-1.patch, jetty1342-20security.patch, mapred2529-trunk.patch We are seeing many instances of the Jetty-1342 (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to stop responding altogether, some fetches go through but a lot of them throw exceptions and eventually fail. The only way we have found to get the TT out of this state is to restart the TT. This jira is to catch this particular exception (or perhaps a configurable regex) and handle it in an automated way to either blacklist or shutdown the TT after seeing it a configurable number of them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2558) Add queue-level metrics 0.20-security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2558: - Fix Version/s: 0.20.204.0 Add queue-level metrics 0.20-security branch Key: MAPREDUCE-2558 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2558 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.20.204.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.20.204.0, 0.20.205.0 Attachments: MAPREDUCE-2558-0.20s.patch, MAPREDUCE-2558-testfixes.patch, queue-metrics-v2.patch, queue-metrics-v3.patch, queue-metrics.patch We would like to record and present the jobtracker metrics on a per-queue basis. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2490: - Fix Version/s: 0.20.204.0 Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.204.0, 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2456) Show the reducer taskid and map/reduce tasktrackers for Failed fetch notification #_ for task attempt... log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2456: - Fix Version/s: 0.20.204.0 Show the reducer taskid and map/reduce tasktrackers for Failed fetch notification #_ for task attempt... log messages --- Key: MAPREDUCE-2456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2456 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Priority: Trivial Fix For: 0.20.204.0, 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2456-0.20s.patch, MAPREDUCE-2456-trunk.patch, MAPREDUCE-2456-trunk.patch Original Estimate: 24h Remaining Estimate: 24h This jira is to provide more useful log information for debugging the Too many fetch-failures error. Looking at the JobTracker node, we see messages like this: 2010-12-14 00:00:06,911 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #8 for task attempt_201011300729_189729_m_007458_0. I would be useful to see which reducer is reporting the error here. So, I propose we add the following to these log messages: 1. reduce task ID 2. TaskTracker nodenames for both the mapper and the reducer -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2479) Backport MAPREDUCE-1568 to hadoop security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2479: - Fix Version/s: 0.20.204.0 Backport MAPREDUCE-1568 to hadoop security branch - Key: MAPREDUCE-2479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2479 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.20.204.0, 0.20.205.0 Attachments: MAPREDUCE-2479-v1.patch, MAPREDUCE-2479-v2.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2451) Log the reason string of healthcheck script
[ https://issues.apache.org/jira/browse/MAPREDUCE-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2451: - Fix Version/s: 0.20.204.0 Log the reason string of healthcheck script --- Key: MAPREDUCE-2451 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2451 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.204.0, 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Trivial Fix For: 0.20.204.0, 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2451-20.205.patch, MAPREDUCE-2451-trunk.patch, MAPREDUCE-2451-trunk.patch Original Estimate: 24h Remaining Estimate: 24h The information on why a specific TaskTracker got blacklisted is not stored anywhere. The jobtracker web ui will show the detailed reason string until the TT gets unblacklisted. After that it is lost. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2601) Add a filter text box to FairSchedulerServlet page
[ https://issues.apache.org/jira/browse/MAPREDUCE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Schober updated MAPREDUCE-2601: -- Attachment: patchfile2.txt Patch file properly generated with --no-prefix. Add a filter text box to FairSchedulerServlet page -- Key: MAPREDUCE-2601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2601 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 0.20.1 Reporter: Scott Chen Priority: Minor Attachments: patchfile.txt, patchfile2.txt It will be useful if we can filter pool in the fairscheduler UI page. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2601) Add a filter text box to FairSchedulerServlet page
[ https://issues.apache.org/jira/browse/MAPREDUCE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051221#comment-13051221 ] Hadoop QA commented on MAPREDUCE-2601: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482961/patchfile2.txt against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/405//console This message is automatically generated. Add a filter text box to FairSchedulerServlet page -- Key: MAPREDUCE-2601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2601 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 0.20.1 Reporter: Scott Chen Priority: Minor Attachments: patchfile.txt, patchfile2.txt It will be useful if we can filter pool in the fairscheduler UI page. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2127) mapreduce trunk builds are filing on hudson ..
[ https://issues.apache.org/jira/browse/MAPREDUCE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051242#comment-13051242 ] Todd Lipcon commented on MAPREDUCE-2127: I'm checking with Nigel to see if we can put this in 0.22 as well, since it's just a build fix and the current issue blocks compilation on some platforms. mapreduce trunk builds are filing on hudson .. --- Key: MAPREDUCE-2127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2127 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 0.20.203.1, 0.20.204.0, 0.23.0 Reporter: Giridharan Kesavan Attachments: MAPREDUCE-2127.patch, MAPREDUCE-2127.patch https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lssl... no [exec] configure: error: Cannot find libssl.so [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2127) mapreduce trunk builds are filing on hudson ..
[ https://issues.apache.org/jira/browse/MAPREDUCE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051241#comment-13051241 ] Todd Lipcon commented on MAPREDUCE-2127: +1, thanks Bruno! mapreduce trunk builds are filing on hudson .. --- Key: MAPREDUCE-2127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2127 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 0.20.203.1, 0.20.204.0, 0.23.0 Reporter: Giridharan Kesavan Attachments: MAPREDUCE-2127.patch, MAPREDUCE-2127.patch https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lssl... no [exec] configure: error: Cannot find libssl.so [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2127) mapreduce trunk builds are filing on hudson ..
[ https://issues.apache.org/jira/browse/MAPREDUCE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned MAPREDUCE-2127: -- Assignee: Bruno Mahé mapreduce trunk builds are filing on hudson .. --- Key: MAPREDUCE-2127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2127 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 0.20.203.1, 0.20.204.0, 0.23.0 Reporter: Giridharan Kesavan Assignee: Bruno Mahé Attachments: MAPREDUCE-2127.patch, MAPREDUCE-2127.patch https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lssl... no [exec] configure: error: Cannot find libssl.so [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051277#comment-13051277 ] Hadoop QA commented on MAPREDUCE-2550: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482952/MAPREDUCE-2550-2.patch against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//console This message is automatically generated. bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550-2.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2558) Add queue-level metrics 0.20-security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2558: - Fix Version/s: (was: 0.20.205.0) Add queue-level metrics 0.20-security branch Key: MAPREDUCE-2558 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2558 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.20.204.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.20.204.0 Attachments: MAPREDUCE-2558-0.20s.patch, MAPREDUCE-2558-testfixes.patch, queue-metrics-v2.patch, queue-metrics-v3.patch, queue-metrics.patch We would like to record and present the jobtracker metrics on a per-queue basis. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2514) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2514: - Fix Version/s: (was: 0.20.205.0) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log Key: MAPREDUCE-2514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2514 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.204.0, 0.23.0 Attachments: MAPREDUCE-2514-branch-0.20-security.patch, MAPREDUCE-2514-trunk.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2524: - Fix Version/s: (was: 0.20.205.0) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle -- Key: MAPREDUCE-2524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Fix For: 0.20.204.0 Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2495) The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2495: - Fix Version/s: (was: 0.20.205.0) The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason - Key: MAPREDUCE-2495 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2495 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Affects Versions: 0.21.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Minor Fix For: 0.20.204.0, 0.23.0 Attachments: MAPREDUCE-2495-20.20X-V1.patch, MAPREDUCE-2495-20.20X-V2.patch, MAPREDUCE-2495-20.20X-V3.patch, MAPREDUCE-2495-20.20X-V4.patch, MAPREDUCE-2495-v1.patch, MAPREDUCE-2495-v2.patch, MAPREDUCE-2495-v3.patch, MAPREDUCE-2495-v4.patch The cleanup thread in the distributed cache handles IOExceptions and the like correctly, but just to be a bit more defensive it would be good to monitor the thread, and check that it is still alive regularly, so that the distributed cache does not fill up the entire disk on the node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2490: - Fix Version/s: (was: 0.20.205.0) Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.204.0, 0.23.0 Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2479) Backport MAPREDUCE-1568 to hadoop security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2479: - Fix Version/s: (was: 0.20.205.0) Backport MAPREDUCE-1568 to hadoop security branch - Key: MAPREDUCE-2479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2479 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.20.204.0 Attachments: MAPREDUCE-2479-v1.patch, MAPREDUCE-2479-v2.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2456) Show the reducer taskid and map/reduce tasktrackers for Failed fetch notification #_ for task attempt... log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2456: - Fix Version/s: (was: 0.20.205.0) Show the reducer taskid and map/reduce tasktrackers for Failed fetch notification #_ for task attempt... log messages --- Key: MAPREDUCE-2456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2456 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Priority: Trivial Fix For: 0.20.204.0, 0.23.0 Attachments: MAPREDUCE-2456-0.20s.patch, MAPREDUCE-2456-trunk.patch, MAPREDUCE-2456-trunk.patch Original Estimate: 24h Remaining Estimate: 24h This jira is to provide more useful log information for debugging the Too many fetch-failures error. Looking at the JobTracker node, we see messages like this: 2010-12-14 00:00:06,911 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #8 for task attempt_201011300729_189729_m_007458_0. I would be useful to see which reducer is reporting the error here. So, I propose we add the following to these log messages: 1. reduce task ID 2. TaskTracker nodenames for both the mapper and the reducer -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2451) Log the reason string of healthcheck script
[ https://issues.apache.org/jira/browse/MAPREDUCE-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-2451: - Fix Version/s: (was: 0.20.205.0) Log the reason string of healthcheck script --- Key: MAPREDUCE-2451 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2451 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.204.0, 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Trivial Fix For: 0.20.204.0, 0.23.0 Attachments: MAPREDUCE-2451-20.205.patch, MAPREDUCE-2451-trunk.patch, MAPREDUCE-2451-trunk.patch Original Estimate: 24h Remaining Estimate: 24h The information on why a specific TaskTracker got blacklisted is not stored anywhere. The jobtracker web ui will show the detailed reason string until the TT gets unblacklisted. After that it is lost. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2569) MR-279: Restarting resource manager with root capacity not equal to 100 percent should result in error
[ https://issues.apache.org/jira/browse/MAPREDUCE-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2569: - Fix Version/s: 0.23.0 MR-279: Restarting resource manager with root capacity not equal to 100 percent should result in error -- Key: MAPREDUCE-2569 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2569 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.0 Attachments: MAPREDUCE-2569-MR-279-v1.patch, MAPREDUCE-2569-MR-279-v2.patch root.capacity is set to 90% without failure -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051318#comment-13051318 ] Todd Lipcon commented on MAPREDUCE-2550: +1, I'll check this into trunk momentarily. bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550-2.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051328#comment-13051328 ] Hudson commented on MAPREDUCE-2550: --- Integrated in Hadoop-Mapreduce-trunk-Commit #730 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/730/]) MAPREDUCE-2550. Fix bin/mapred to work properly from within a source checkout. Contributed by Eric Yang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1137017 Files : * /hadoop/common/trunk/mapreduce/bin/mapred-config.sh * /hadoop/common/trunk/mapreduce/CHANGES.txt * /hadoop/common/trunk/mapreduce/bin/start-mapred.sh * /hadoop/common/trunk/mapreduce/bin/stop-mapred.sh * /hadoop/common/trunk/mapreduce/bin/mapred bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550-2.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1638) Divide MapReduce into API and implementation source trees
[ https://issues.apache.org/jira/browse/MAPREDUCE-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051382#comment-13051382 ] Todd Lipcon commented on MAPREDUCE-1638: Could the toFullPropertyName method be moved to MRJobConfig and then used by both QueueManager and JobSubmitter? Duplicating it seems ugly. Divide MapReduce into API and implementation source trees - Key: MAPREDUCE-1638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1638 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-1638.patch, MAPREDUCE-1638.patch, MAPREDUCE-1638.sh I think it makes sense to separate the MapReduce source into public API and implementation trees. The public API could be broken further into kernel and library trees. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2421) Remove JobHistory's dependency on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051385#comment-13051385 ] Todd Lipcon commented on MAPREDUCE-2421: hrm, moving convertTrackerNameToHostName and getTaskLogUrl to HistoryViewer seems somewhat misplaced. Is there no better place that it can go that's OK from a dependency perspective? Remove JobHistory's dependency on JobTracker Key: MAPREDUCE-2421 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2421 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-2421.patch, MAPREDUCE-2421.patch, MAPREDUCE-2421.patch The direct dependency can be removed by introducing a callback inteface. This is for MAPREDUCE-1638. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2323) Add metrics to the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2323: --- Attachment: mr-2323.txt Rebased on trunk. Also fixes a bug where using setPool twice on a job would cause an NPE, since it would clean up the metrics on the first switch, and then NPE when cleaning it up again Add metrics to the fair scheduler - Key: MAPREDUCE-2323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2323) Add metrics to the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051449#comment-13051449 ] Hadoop QA commented on MAPREDUCE-2323: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483015/mr-2323.txt against trunk revision 1137017. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//console This message is automatically generated. Add metrics to the fair scheduler - Key: MAPREDUCE-2323 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
