[jira] [Commented] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354398#comment-14354398 ] Hadoop QA commented on MAPREDUCE-5916: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652404/MAPREDUCE-5916.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5272//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5272//console This message is automatically generated. The authenticate response is not sent when password is empty (LocalJobRunner) - Key: MAPREDUCE-5916 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Reporter: Joe Mudd Attachments: MAPREDUCE-5916.patch When running in a mode where there are no credentials associated with the pipes submission and the password is empty, the C++ verifyDigestAndRespond() does not respond to the Java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6263: -- Summary: Configurable timeout between YARNRunner terminate the application and forcefully kill. (was: Large jobs can lose history when killed due to brief client timeout) Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6263: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have commit v2 patch to trunk, branch-2 and branch-2.7. Thanks [~eepayne] for the contribution! Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354858#comment-14354858 ] Eric Payne commented on MAPREDUCE-6263: --- Thank you, [~djp] Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Labels: perfomance (was: ) FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) Speed up FileOutputCommitter#commitJob for many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Summary: Speed up FileOutputCommitter#commitJob for many output files (was: FileOutputCommitter.commitJob can be very slow for jobs with many output files) Speed up FileOutputCommitter#commitJob for many output files Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Issue Type: Improvement (was: Bug) FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355308#comment-14355308 ] Gera Shegalov commented on MAPREDUCE-4815: -- Committing v17. FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4742) Tiny nnbench displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355083#comment-14355083 ] Tsuyoshi Ozawa commented on MAPREDUCE-4742: --- +1, the test failure is not related to the patch since it only includes fixes about comments. Committing this shortly. Tiny nnbench displayUsage - Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Liang Xie Priority: Trivial Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4742) Fix typo in nnbench#displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4742: -- Summary: Fix typo in nnbench#displayUsage (was: Tiny nnbench displayUsage) Fix typo in nnbench#displayUsage Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4, 2.6.0 Reporter: Liang Xie Priority: Trivial Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4742) Fix typo in nnbench#displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355110#comment-14355110 ] Hudson commented on MAPREDUCE-4742: --- FAILURE: Integrated in Hadoop-trunk-Commit #7297 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7297/]) MAPREDUCE-4742. Fix typo in nnbench#displayUsage. Contributed by Liang Xie. (ozawa: rev 20b8ee1350e62d1b21c951e653302b6e6a8e4f7e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java * hadoop-mapreduce-project/CHANGES.txt Fix typo in nnbench#displayUsage Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4, 2.6.0 Reporter: Liang Xie Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) Speed up FileOutputCommitter#commitJob for many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~l201514] for working through all review cycles, and [~mingma], [~dlmarion], [~shrijeet], and last but not least [~jlowe] for additional reviews. Committed to trunk, branch-2, and branch-2.7. Speed up FileOutputCommitter#commitJob for many output files Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Fix For: 2.7.0 Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6272) NodeManager should use directory other than tmp for extracting and loading leveldbjni
Anubhav Dhoot created MAPREDUCE-6272: Summary: NodeManager should use directory other than tmp for extracting and loading leveldbjni Key: MAPREDUCE-6272 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6272 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot /tmp can be required to be noexec in many environments. This causes a problem when nodemanager tries to load the leveldbjni library which can get unpacked and executed from /tmp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4919) All maps hangs when set mapreduce.task.io.sort.factor to 1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354443#comment-14354443 ] Hadoop QA commented on MAPREDUCE-4919: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563721/MAPREDUCE-4919.patch against trunk revision 47f7f18. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestLineRecordReader org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.v2.TestMROldApiJobs The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.output.TestJobTests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestMRIntermediateDataEncryption Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5261//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5261//console This message is automatically generated. All maps hangs when set mapreduce.task.io.sort.factor to 1 -- Key: MAPREDUCE-4919 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4919 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Jerry Chen Assignee: Jerry Chen Attachments: MAPREDUCE-4919.patch Original Estimate: 2h Remaining Estimate: 2h In one of my testing that when I set mapreduce.task.io.sort.factor to 1, all the maps hang and will never end. But the CPU usage for each node are very high and until killed by the app master when time out comes, and the job failed. I traced the problem and found out that all the maps hangs on the final merge phase. The while loop in computeBytesInMerges will never end with a factor of 1: int f = 1; //in my case int n = 16; //in my case while (n f || considerFinalMerge) { ... n -= (f-1); f = factor; } As the f-1 will equals 0 and n will always be 16 and the while runs for ever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354445#comment-14354445 ] Hadoop QA commented on MAPREDUCE-1362: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652261/MAPREDUCE-1362.patch against trunk revision 47f7f18. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.TestChild Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5263//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5263//console This message is automatically generated. Pipes should be ported to the new mapreduce API --- Key: MAPREDUCE-1362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 Project: Hadoop Map/Reduce Issue Type: Improvement Components: pipes Reporter: Bassam Tabbara Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, MAPREDUCE-1362.patch Pipes is still currently using the old mapred API. This prevents us from using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. Here is a rough proposal for how to accomplish this: * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new mapred API. * the new pipes package will run side by side with the old one. old one should get deprecated at some point. * the wire protocol used between PipesMapper and PipesReducer and C++ programs must not change. * bin/hadoop should support both pipes (old api) and pipes2 (new api) Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354455#comment-14354455 ] Hadoop QA commented on MAPREDUCE-5917: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652405/MAPREDUCE-5917.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5278//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5278//console This message is automatically generated. Be able to retrieve configuration keys by index --- Key: MAPREDUCE-5917 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917 Project: Hadoop Map/Reduce Issue Type: New Feature Components: pipes Reporter: Joe Mudd Priority: Minor Attachments: MAPREDUCE-5917.patch The pipes C++ side does not have a configuration key/value pair iterator. It is useful to be able to iterate through all of the configuration keys without having to expose a C++ map iterator since that is specific to the JobConf internals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5491) DFSIO do not initialize write buffer correctly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354452#comment-14354452 ] Hadoop QA commented on MAPREDUCE-5491: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12602080/MAPREDUCE-5491-v2.patch against trunk revision 47f7f18. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestMRTimelineEventHandling Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5268//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5268//console This message is automatically generated. DFSIO do not initialize write buffer correctly -- Key: MAPREDUCE-5491 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5491 Project: Hadoop Map/Reduce Issue Type: Bug Components: benchmarks, test Reporter: Raymond Liu Assignee: Raymond Liu Attachments: MAPREDUCE-5491-v2.patch, MAPREDUCE-5491.patch In DFSIO test, the IOMapperBase will set bufferSize in configure method, while writeMapper, appendMapper etc use bufferSize to initialize buffer in the constructor. This will lead to buffer not initialized at all. It is ok for non compression route, while compression is used, the output data size will be very small due to all 0 in buffer. Thus, the overrided configure method should be be the correct place for initial buffer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354414#comment-14354414 ] Hadoop QA commented on MAPREDUCE-5860: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662748/MAPREDUCE-5860.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5274//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5274//console This message is automatically generated. Hadoop pipes Combiner is closed before all of its reduce calls -- Key: MAPREDUCE-5860 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.0 Environment: 0.23.0 on 64 bit linux Reporter: Joe Mudd Attachments: MAPREDUCE-5860.patch When a Combiner is specified to runTask() its reduce() method may be called after its close() method has been called due to how the Combiner's containing object, CombineRunner, is closed after the TaskContextImpl's reducer member is closed (see TaskContextImpl::closeAll()). I believe the fix is to delegate the Combiner's ownership to CombineRunner, making it responsible for calling the Combiner's close() method and deleting the Combiner instance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4710) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354421#comment-14354421 ] Hadoop QA commented on MAPREDUCE-4710: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620887/mapreduce4710-v6.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5277//console This message is automatically generated. Add peak memory usage counter for each task --- Key: MAPREDUCE-4710 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4710 Project: Hadoop Map/Reduce Issue Type: New Feature Components: task Affects Versions: 1.0.2 Reporter: Cindy Li Assignee: Cindy Li Priority: Minor Labels: patch Attachments: MAPREDUCE-4710-trunk.patch, mapreduce-4710-v1.0.2.patch, mapreduce-4710.patch, mapreduce4710-v3.patch, mapreduce4710-v6.patch, mapreduce4710.patch Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which are snapshots of memory usage of that task. They are not sufficient for users to understand peak memory usage by that task, e.g. in order to diagnose task failures, tune job parameters or change application design. This new feature will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and VIRTUAL_MEMORY_BYTES_MAX. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3097) archive does not archive if the content specified is a file
[ https://issues.apache.org/jira/browse/MAPREDUCE-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354423#comment-14354423 ] Hadoop QA commented on MAPREDUCE-3097: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12590322/MAPREDUCE-3097.patch against trunk revision 7711049. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5279//console This message is automatically generated. archive does not archive if the content specified is a file --- Key: MAPREDUCE-3097 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3097 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.203.0, 0.20.205.0 Reporter: Arpit Gupta Priority: Minor Attachments: MAPREDUCE-3097.patch archive command only archives directories. when the content specified is a file it proceeds with the archive job but does not archive the content this can be misleading as the user might think that archive was successful. We should change it to either throw an error or make it archive files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354437#comment-14354437 ] Hadoop QA commented on MAPREDUCE-3914: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652401/MAPREDUCE-3914.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5276//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5276//console This message is automatically generated. Mismatched free() / delete / delete [] in HadoopPipes - Key: MAPREDUCE-3914 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 Reporter: Charles Earl Attachments: MAPREDUCE-3914-branch-0.23.patch, MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch Original Estimate: 1h Remaining Estimate: 1h When running valgrind on a simple MapReduce pipes job, valgrind identifies a mismatched new / delete: ==20394== Mismatched free() / delete / delete [] ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1171) ==20394==by 0x424C33: main (ProcessRow.cpp:118) ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd ==20394==at 0x4C2864B: operator new[](unsigned long) (vg_replace_malloc.c:305) ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1121) ==20394==by 0x424C33: main (ProcessRow.cpp:118) ==20394== ==20394== Mismatched free() / delete / delete [] ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1172) ==20394==by 0x424C33: main (ProcessRow.cpp:118) ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd ==20394==at 0x4C2864B: operator new[](unsigned long) (vg_replace_malloc.c:305) ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const) (HadoopPipes.cc:1122) ==20394==by 0x424C33: main (ProcessRow.cpp:118) The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: bufin = new char[bufsize]; bufout = new char[bufsize]; should have matching delete [] calls but are instead bracketed my delete on lines 1171 and 1172: delete bufin; delete bufout; So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5869) Wrong date and and time on job tracker page
[ https://issues.apache.org/jira/browse/MAPREDUCE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354408#comment-14354408 ] Hadoop QA commented on MAPREDUCE-5869: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642534/WRONGDATE_TIME.patch against trunk revision 7711049. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5275//console This message is automatically generated. Wrong date and and time on job tracker page --- Key: MAPREDUCE-5869 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5869 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.2.0 Reporter: chaitali gupta Assignee: chaitali gupta Priority: Trivial Attachments: WRONGDATE_TIME.patch When an application master restarts during execution of a task, job tracker page displays wrong start date-time for the job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4363) Hadoop 1.X, 2.X and trunk do not build on Fedora 17
[ https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354409#comment-14354409 ] Hadoop QA commented on MAPREDUCE-4363: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533137/MAPREDUCE-4363-trunk.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5273//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5273//console This message is automatically generated. Hadoop 1.X, 2.X and trunk do not build on Fedora 17 --- Key: MAPREDUCE-4363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, pipes Affects Versions: 1.0.3 Reporter: Bruno Mahé Assignee: Bruno Mahé Labels: bigtop Attachments: MAPREDUCE-4363-trunk.patch, MAPREDUCE-4363.patch I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is failing to build. This seems related to the bump in version of gcc to 4.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4742) Tiny nnbench displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354432#comment-14354432 ] Hadoop QA commented on MAPREDUCE-4742: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550422/MAPREDUCE-4742.txt against trunk revision 47f7f18. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestMRCJCFileInputFormat The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestJobCleanup Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5262//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5262//console This message is automatically generated. Tiny nnbench displayUsage - Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Liang Xie Priority: Trivial Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355769#comment-14355769 ] Allen Wittenauer commented on MAPREDUCE-6246: - I think a test here would be good. DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2 - Key: MAPREDUCE-6246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.4.1 Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x Platform: xSeries, pSeries Browser: Firefox, IE Security Settings: No Security, Flat file, LDAP, PAM File System: HDFS, GPFS FPO Reporter: ramtin Assignee: ramtin Labels: DB2, mapreduce Attachments: MAPREDUCE-6246.patch Original Estimate: 24h Remaining Estimate: 24h DBoutputformat is used for writing output of mapreduce jobs to the database and when used with db2 jdbc drivers it fails with following error com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT, DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127) In DBOutputFormat class there is constructQuery method that generates INSERT INTO statement with semicolon(;) at the end. Semicolon is ANSI SQL-92 standard character for a statement terminator but this feature is disabled(OFF) as a default settings in IBM DB2. Although by using -t we can turn it ON for db2. (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2). But there are some products that already built on top of this default setting (OFF) so by turning ON this feature make them error prone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6222) HistoryServer Hangs Processing Large Jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354507#comment-14354507 ] Hadoop QA commented on MAPREDUCE-6222: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703589/MAPREDUCE-6222.003.patch against trunk revision 7711049. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5280//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5280//console This message is automatically generated. HistoryServer Hangs Processing Large Jobs - Key: MAPREDUCE-6222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6222 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Andrew Johnson Assignee: Ray Chiang Attachments: JHS New Display Top.png, JHS Original Display Top.png, MAPREDUCE-6222.001.patch, MAPREDUCE-6222.002.patch, MAPREDUCE-6222.003.patch, head.jhist, historyserver_jstack.txt I'm encountering an issue with the Mapreduce HistoryServer processing the history files for large jobs. This has come up several times with for jobs with around 6 total tasks. When the HistoryServer loads the .jhist file from HDFS for a job of that size (which is usually around 500 Mb), the HistoryServer's CPU usage spiked and the UI became unresponsive. After about 10 minutes I restarted the HistoryServer and it was behaving normally again. The cluster is running CDH 5.3 (2.5.0-cdh5.3.0). I've attached the output of jstack from a time this was occurring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4424) The 'mapred job -list' command should show the job name as well
[ https://issues.apache.org/jira/browse/MAPREDUCE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354503#comment-14354503 ] Hadoop QA commented on MAPREDUCE-4424: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592884/MAPREDUCE-4424-3.patch against trunk revision 47f7f18. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader org.apache.hadoop.mapred.TestLineRecordReader org.apache.hadoop.mapred.TestMiniMRClientCluster Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5270//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5270//console This message is automatically generated. The 'mapred job -list' command should show the job name as well --- Key: MAPREDUCE-4424 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4424 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Harsh J Assignee: Avinash Kujur Priority: Trivial Labels: newbie Attachments: MAPREDUCE-4424-2.patch, MAPREDUCE-4424-3.patch, MAPREDUCE-4424.patch Currently the {{mapred job -list}} command does not show the Job Name, just the Job ID. It would be good to display the Job name too. Idea originally from HADOOP-. -- This message was sent by Atlassian JIRA (v6.3.4#6332)