[jira] [Commented] (MAPREDUCE-5916) The authenticate response is not sent when password is empty (LocalJobRunner)
[
https://issues.apache.org/jira/browse/MAPREDUCE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354398#comment-14354398
]
Hadoop QA commented on MAPREDUCE-5916:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652404/MAPREDUCE-5916.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5272//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5272//console
This message is automatically generated.
The authenticate response is not sent when password is empty (LocalJobRunner)
-
Key: MAPREDUCE-5916
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5916
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Reporter: Joe Mudd
Attachments: MAPREDUCE-5916.patch
When running in a mode where there are no credentials associated with the
pipes submission and the password is empty, the C++ verifyDigestAndRespond()
does not respond to the Java side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6263: -- Summary: Configurable timeout between YARNRunner terminate the application and forcefully kill. (was: Large jobs can lose history when killed due to brief client timeout) Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6263: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have commit v2 patch to trunk, branch-2 and branch-2.7. Thanks [~eepayne] for the contribution! Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354858#comment-14354858 ] Eric Payne commented on MAPREDUCE-6263: --- Thank you, [~djp] Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Labels: perfomance (was: ) FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) Speed up FileOutputCommitter#commitJob for many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Summary: Speed up FileOutputCommitter#commitJob for many output files (was: FileOutputCommitter.commitJob can be very slow for jobs with many output files) Speed up FileOutputCommitter#commitJob for many output files Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Issue Type: Improvement (was: Bug) FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355308#comment-14355308 ] Gera Shegalov commented on MAPREDUCE-4815: -- Committing v17. FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4742) Tiny nnbench displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355083#comment-14355083 ] Tsuyoshi Ozawa commented on MAPREDUCE-4742: --- +1, the test failure is not related to the patch since it only includes fixes about comments. Committing this shortly. Tiny nnbench displayUsage - Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Liang Xie Priority: Trivial Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4742) Fix typo in nnbench#displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4742: -- Summary: Fix typo in nnbench#displayUsage (was: Tiny nnbench displayUsage) Fix typo in nnbench#displayUsage Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4, 2.6.0 Reporter: Liang Xie Priority: Trivial Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4742) Fix typo in nnbench#displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355110#comment-14355110 ] Hudson commented on MAPREDUCE-4742: --- FAILURE: Integrated in Hadoop-trunk-Commit #7297 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7297/]) MAPREDUCE-4742. Fix typo in nnbench#displayUsage. Contributed by Liang Xie. (ozawa: rev 20b8ee1350e62d1b21c951e653302b6e6a8e4f7e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java * hadoop-mapreduce-project/CHANGES.txt Fix typo in nnbench#displayUsage Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4, 2.6.0 Reporter: Liang Xie Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) Speed up FileOutputCommitter#commitJob for many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4815: - Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~l201514] for working through all review cycles, and [~mingma], [~dlmarion], [~shrijeet], and last but not least [~jlowe] for additional reviews. Committed to trunk, branch-2, and branch-2.7. Speed up FileOutputCommitter#commitJob for many output files Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Fix For: 2.7.0 Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6272) NodeManager should use directory other than tmp for extracting and loading leveldbjni
Anubhav Dhoot created MAPREDUCE-6272: Summary: NodeManager should use directory other than tmp for extracting and loading leveldbjni Key: MAPREDUCE-6272 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6272 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot /tmp can be required to be noexec in many environments. This causes a problem when nodemanager tries to load the leveldbjni library which can get unpacked and executed from /tmp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4919) All maps hangs when set mapreduce.task.io.sort.factor to 1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354443#comment-14354443
]
Hadoop QA commented on MAPREDUCE-4919:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563721/MAPREDUCE-4919.patch
against trunk revision 47f7f18.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestLineRecordReader
org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader
org.apache.hadoop.mapred.TestMRTimelineEventHandling
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.lib.output.TestJobTests
org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5261//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5261//console
This message is automatically generated.
All maps hangs when set mapreduce.task.io.sort.factor to 1
--
Key: MAPREDUCE-4919
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4919
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Jerry Chen
Assignee: Jerry Chen
Attachments: MAPREDUCE-4919.patch
Original Estimate: 2h
Remaining Estimate: 2h
In one of my testing that when I set mapreduce.task.io.sort.factor to 1, all
the maps hang and will never end. But the CPU usage for each node are very
high and until killed by the app master when time out comes, and the job
failed.
I traced the problem and found out that all the maps hangs on the final merge
phase.
The while loop in computeBytesInMerges will never end with a factor of 1:
int f = 1; //in my case
int n = 16; //in my case
while (n f || considerFinalMerge) {
...
n -= (f-1);
f = factor;
}
As the f-1 will equals 0 and n will always be 16 and the while runs for ever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[
https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354445#comment-14354445
]
Hadoop QA commented on MAPREDUCE-1362:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652261/MAPREDUCE-1362.patch
against trunk revision 47f7f18.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapreduce.TestChild
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5263//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5263//console
This message is automatically generated.
Pipes should be ported to the new mapreduce API
---
Key: MAPREDUCE-1362
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: pipes
Reporter: Bassam Tabbara
Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch,
MAPREDUCE-1362.patch
Pipes is still currently using the old mapred API. This prevents us from
using pipes with HBase's TableInputFormat, HRegionPartitioner, etc.
Here is a rough proposal for how to accomplish this:
* Add a new package org.apache.hadoop.mapreduce.pipes that uses the new
mapred API.
* the new pipes package will run side by side with the old one. old one
should get deprecated at some point.
* the wire protocol used between PipesMapper and PipesReducer and C++
programs must not change.
* bin/hadoop should support both pipes (old api) and pipes2 (new api)
Does this sound reasonable?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5917) Be able to retrieve configuration keys by index
[
https://issues.apache.org/jira/browse/MAPREDUCE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354455#comment-14354455
]
Hadoop QA commented on MAPREDUCE-5917:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652405/MAPREDUCE-5917.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5278//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5278//console
This message is automatically generated.
Be able to retrieve configuration keys by index
---
Key: MAPREDUCE-5917
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5917
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: pipes
Reporter: Joe Mudd
Priority: Minor
Attachments: MAPREDUCE-5917.patch
The pipes C++ side does not have a configuration key/value pair iterator. It
is useful to be able to iterate through all of the configuration keys without
having to expose a C++ map iterator since that is specific to the JobConf
internals.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5491) DFSIO do not initialize write buffer correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-5491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354452#comment-14354452
]
Hadoop QA commented on MAPREDUCE-5491:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602080/MAPREDUCE-5491-v2.patch
against trunk revision 47f7f18.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestMRTimelineEventHandling
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5268//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5268//console
This message is automatically generated.
DFSIO do not initialize write buffer correctly
--
Key: MAPREDUCE-5491
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5491
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks, test
Reporter: Raymond Liu
Assignee: Raymond Liu
Attachments: MAPREDUCE-5491-v2.patch, MAPREDUCE-5491.patch
In DFSIO test, the IOMapperBase will set bufferSize in configure method,
while writeMapper, appendMapper etc use bufferSize to initialize buffer in
the constructor. This will lead to buffer not initialized at all. It is ok
for non compression route, while compression is used, the output data size
will be very small due to all 0 in buffer.
Thus, the overrided configure method should be be the correct place for
initial buffer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354414#comment-14354414
]
Hadoop QA commented on MAPREDUCE-5860:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662748/MAPREDUCE-5860.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5274//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5274//console
This message is automatically generated.
Hadoop pipes Combiner is closed before all of its reduce calls
--
Key: MAPREDUCE-5860
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 0.23.0
Environment: 0.23.0 on 64 bit linux
Reporter: Joe Mudd
Attachments: MAPREDUCE-5860.patch
When a Combiner is specified to runTask() its reduce() method may be called
after its close() method has been called due to how the Combiner's containing
object, CombineRunner, is closed after the TaskContextImpl's reducer member
is closed (see TaskContextImpl::closeAll()).
I believe the fix is to delegate the Combiner's ownership to CombineRunner,
making it responsible for calling the Combiner's close() method and deleting
the Combiner instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4710) Add peak memory usage counter for each task
[
https://issues.apache.org/jira/browse/MAPREDUCE-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354421#comment-14354421
]
Hadoop QA commented on MAPREDUCE-4710:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620887/mapreduce4710-v6.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5277//console
This message is automatically generated.
Add peak memory usage counter for each task
---
Key: MAPREDUCE-4710
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4710
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: task
Affects Versions: 1.0.2
Reporter: Cindy Li
Assignee: Cindy Li
Priority: Minor
Labels: patch
Attachments: MAPREDUCE-4710-trunk.patch, mapreduce-4710-v1.0.2.patch,
mapreduce-4710.patch, mapreduce4710-v3.patch, mapreduce4710-v6.patch,
mapreduce4710.patch
Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which
are snapshots of memory usage of that task. They are not sufficient for users
to understand peak memory usage by that task, e.g. in order to diagnose task
failures, tune job parameters or change application design. This new feature
will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and
VIRTUAL_MEMORY_BYTES_MAX.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3097) archive does not archive if the content specified is a file
[
https://issues.apache.org/jira/browse/MAPREDUCE-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354423#comment-14354423
]
Hadoop QA commented on MAPREDUCE-3097:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590322/MAPREDUCE-3097.patch
against trunk revision 7711049.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5279//console
This message is automatically generated.
archive does not archive if the content specified is a file
---
Key: MAPREDUCE-3097
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3097
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.203.0, 0.20.205.0
Reporter: Arpit Gupta
Priority: Minor
Attachments: MAPREDUCE-3097.patch
archive command only archives directories. when the content specified is a
file it proceeds with the archive job but does not archive the content this
can be misleading as the user might think that archive was successful. We
should change it to either throw an error or make it archive files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354437#comment-14354437
]
Hadoop QA commented on MAPREDUCE-3914:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652401/MAPREDUCE-3914.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5276//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5276//console
This message is automatically generated.
Mismatched free() / delete / delete [] in HadoopPipes
-
Key: MAPREDUCE-3914
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 0.20.205.0, 0.23.0, 1.0.0
Environment: Based upon map reduce pipes task executed on Ubuntu 11.10
Reporter: Charles Earl
Attachments: MAPREDUCE-3914-branch-0.23.patch,
MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch
Original Estimate: 1h
Remaining Estimate: 1h
When running valgrind on a simple MapReduce pipes job, valgrind identifies a
mismatched new / delete:
==20394== Mismatched free() / delete / delete []
==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387)
==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const)
(HadoopPipes.cc:1171)
==20394==by 0x424C33: main (ProcessRow.cpp:118)
==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd
==20394==at 0x4C2864B: operator new[](unsigned long)
(vg_replace_malloc.c:305)
==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const)
(HadoopPipes.cc:1121)
==20394==by 0x424C33: main (ProcessRow.cpp:118)
==20394==
==20394== Mismatched free() / delete / delete []
==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387)
==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const)
(HadoopPipes.cc:1172)
==20394==by 0x424C33: main (ProcessRow.cpp:118)
==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd
==20394==at 0x4C2864B: operator new[](unsigned long)
(vg_replace_malloc.c:305)
==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const)
(HadoopPipes.cc:1122)
==20394==by 0x424C33: main (ProcessRow.cpp:118)
The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc:
bufin = new char[bufsize];
bufout = new char[bufsize];
should have matching delete [] calls but are instead bracketed my delete on
lines 1171 and 1172:
delete bufin;
delete bufout;
So these should be replaced by delete[]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5869) Wrong date and and time on job tracker page
[
https://issues.apache.org/jira/browse/MAPREDUCE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354408#comment-14354408
]
Hadoop QA commented on MAPREDUCE-5869:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642534/WRONGDATE_TIME.patch
against trunk revision 7711049.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5275//console
This message is automatically generated.
Wrong date and and time on job tracker page
---
Key: MAPREDUCE-5869
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5869
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: chaitali gupta
Assignee: chaitali gupta
Priority: Trivial
Attachments: WRONGDATE_TIME.patch
When an application master restarts during execution of a task, job tracker
page displays wrong start date-time for the job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4363) Hadoop 1.X, 2.X and trunk do not build on Fedora 17
[
https://issues.apache.org/jira/browse/MAPREDUCE-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354409#comment-14354409
]
Hadoop QA commented on MAPREDUCE-4363:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12533137/MAPREDUCE-4363-trunk.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5273//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5273//console
This message is automatically generated.
Hadoop 1.X, 2.X and trunk do not build on Fedora 17
---
Key: MAPREDUCE-4363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, pipes
Affects Versions: 1.0.3
Reporter: Bruno Mahé
Assignee: Bruno Mahé
Labels: bigtop
Attachments: MAPREDUCE-4363-trunk.patch, MAPREDUCE-4363.patch
I upgraded my machine to the latest Fedora 17 and now Apache Hadoop is
failing to build. This seems related to the bump in version of gcc to 4.7.0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4742) Tiny nnbench displayUsage
[
https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354432#comment-14354432
]
Hadoop QA commented on MAPREDUCE-4742:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550422/MAPREDUCE-4742.txt
against trunk revision 47f7f18.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestMRCJCFileInputFormat
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestJobCleanup
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5262//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5262//console
This message is automatically generated.
Tiny nnbench displayUsage
-
Key: MAPREDUCE-4742
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.0.2-alpha, 0.23.4
Reporter: Liang Xie
Priority: Trivial
Attachments: MAPREDUCE-4742.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355769#comment-14355769 ] Allen Wittenauer commented on MAPREDUCE-6246: - I think a test here would be good. DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2 - Key: MAPREDUCE-6246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.4.1 Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x Platform: xSeries, pSeries Browser: Firefox, IE Security Settings: No Security, Flat file, LDAP, PAM File System: HDFS, GPFS FPO Reporter: ramtin Assignee: ramtin Labels: DB2, mapreduce Attachments: MAPREDUCE-6246.patch Original Estimate: 24h Remaining Estimate: 24h DBoutputformat is used for writing output of mapreduce jobs to the database and when used with db2 jdbc drivers it fails with following error com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT, DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127) In DBOutputFormat class there is constructQuery method that generates INSERT INTO statement with semicolon(;) at the end. Semicolon is ANSI SQL-92 standard character for a statement terminator but this feature is disabled(OFF) as a default settings in IBM DB2. Although by using -t we can turn it ON for db2. (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2). But there are some products that already built on top of this default setting (OFF) so by turning ON this feature make them error prone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6222) HistoryServer Hangs Processing Large Jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354507#comment-14354507
]
Hadoop QA commented on MAPREDUCE-6222:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703589/MAPREDUCE-6222.003.patch
against trunk revision 7711049.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5280//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5280//console
This message is automatically generated.
HistoryServer Hangs Processing Large Jobs
-
Key: MAPREDUCE-6222
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6222
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Andrew Johnson
Assignee: Ray Chiang
Attachments: JHS New Display Top.png, JHS Original Display Top.png,
MAPREDUCE-6222.001.patch, MAPREDUCE-6222.002.patch, MAPREDUCE-6222.003.patch,
head.jhist, historyserver_jstack.txt
I'm encountering an issue with the Mapreduce HistoryServer processing the
history files for large jobs. This has come up several times with for jobs
with around 6 total tasks. When the HistoryServer loads the .jhist file
from HDFS for a job of that size (which is usually around 500 Mb), the
HistoryServer's CPU usage spiked and the UI became unresponsive. After about
10 minutes I restarted the HistoryServer and it was behaving normally again.
The cluster is running CDH 5.3 (2.5.0-cdh5.3.0). I've attached the output of
jstack from a time this was occurring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4424) The 'mapred job -list' command should show the job name as well
[
https://issues.apache.org/jira/browse/MAPREDUCE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354503#comment-14354503
]
Hadoop QA commented on MAPREDUCE-4424:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592884/MAPREDUCE-4424-3.patch
against trunk revision 47f7f18.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader
org.apache.hadoop.mapred.TestLineRecordReader
org.apache.hadoop.mapred.TestMiniMRClientCluster
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5270//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5270//console
This message is automatically generated.
The 'mapred job -list' command should show the job name as well
---
Key: MAPREDUCE-4424
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4424
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: Harsh J
Assignee: Avinash Kujur
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-4424-2.patch, MAPREDUCE-4424-3.patch,
MAPREDUCE-4424.patch
Currently the {{mapred job -list}} command does not show the Job Name, just
the Job ID. It would be good to display the Job name too. Idea originally
from HADOOP-.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
