[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Attachment: MAPREDUCE-5718.v2.patch I think we can delete the commit tmp file at init stage MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6141) History server leveldb recovery store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6141: -- Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. History server leveldb recovery store - Key: MAPREDUCE-6141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: MAPREDUCE-6141.patch, MAPREDUCE-6141v2.patch, MAPREDUCE-6141v2.patch It would be nice to have a leveldb option to the job history server recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6141) History server leveldb recovery store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292019#comment-14292019 ] Jason Lowe commented on MAPREDUCE-6141: --- All of the findbug warnings are unrelated, filed MAPREDUCE-6225. Committing this. History server leveldb recovery store - Key: MAPREDUCE-6141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-6141.patch, MAPREDUCE-6141v2.patch, MAPREDUCE-6141v2.patch It would be nice to have a leveldb option to the job history server recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6225) Fix new findbug warnings in hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292018#comment-14292018 ] Jason Lowe commented on MAPREDUCE-6225: --- See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5121//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html as one example of the warnings. Fix new findbug warnings in hadoop-mapreduce-client-core Key: MAPREDUCE-6225 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6225 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Jason Lowe Recent precommit builds in hadoop-mapreduce-client-core are flagging findbug warnings that appear to be new with the recent findbugs upgrade. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6141) History server leveldb recovery store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6141: -- Attachment: MAPREDUCE-6141v2.patch Thanks for the review, Jon! Uploading the same patch again to kick Jenkins since it's been a while. I'll commit this if Jenkins doesn't indicate any problems with the patch. History server leveldb recovery store - Key: MAPREDUCE-6141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-6141.patch, MAPREDUCE-6141v2.patch, MAPREDUCE-6141v2.patch It would be nice to have a leveldb option to the job history server recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6141) History server leveldb recovery store
[
https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291977#comment-14291977
]
Hadoop QA commented on MAPREDUCE-6141:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694558/MAPREDUCE-6141v2.patch
against trunk revision 2b0fa20.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5121//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5121//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5121//console
This message is automatically generated.
History server leveldb recovery store
-
Key: MAPREDUCE-6141
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-6141.patch, MAPREDUCE-6141v2.patch,
MAPREDUCE-6141v2.patch
It would be nice to have a leveldb option to the job history server recovery
store. Leveldb would provide some benefits over the existing filesystem
store such as better support for atomic operations, fewer I/O ops per state
update, and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6225) Fix new findbug warnings in hadoop-mapreduce-client-core
Jason Lowe created MAPREDUCE-6225: - Summary: Fix new findbug warnings in hadoop-mapreduce-client-core Key: MAPREDUCE-6225 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6225 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Jason Lowe Recent precommit builds in hadoop-mapreduce-client-core are flagging findbug warnings that appear to be new with the recent findbugs upgrade. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6141) History server leveldb recovery store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292037#comment-14292037 ] Hudson commented on MAPREDUCE-6141: --- SUCCESS: Integrated in Hadoop-trunk-Commit #6932 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6932/]) MAPREDUCE-6141. History server leveldb recovery store. Contributed by Jason Lowe (jlowe: rev 56b7ec71a69820ae12b4b9e2eb04b7368f721dbf) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestHistoryServerLeveldbStateStoreService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java History server leveldb recovery store - Key: MAPREDUCE-6141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: MAPREDUCE-6141.patch, MAPREDUCE-6141v2.patch, MAPREDUCE-6141v2.patch It would be nice to have a leveldb option to the job history server recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292223#comment-14292223 ] Ravi Prakash commented on MAPREDUCE-6220: - I believe a lot of applications (including oozie) use stdout and stderr for diagnostics. Unless you can think of a better way to expose these messages I don't agree we should do this. Closing as Won't Fix To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash resolved MAPREDUCE-6220. - Resolution: Not a Problem To forbid stderr and stdout for MapReduce job - Key: MAPREDUCE-6220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6220.patch System.out and System.error is a ugly way to print log, and many times it would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6226) Findbugs warnings in mapreduce-client-core
Varun Saxena created MAPREDUCE-6226: --- Summary: Findbugs warnings in mapreduce-client-core Key: MAPREDUCE-6226 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6226 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Varun Saxena Assignee: Varun Saxena Attachments: newPatchFindbugsWarningshadoop-mapreduce-client-core.html Found findbugs warnings reported for mapreduce-client-core while fixing MAPREDUCE-6223 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6226) Findbugs warnings in mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-6226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6226: Attachment: newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings in mapreduce-client-core -- Key: MAPREDUCE-6226 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6226 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Varun Saxena Assignee: Varun Saxena Attachments: newPatchFindbugsWarningshadoop-mapreduce-client-core.html Found findbugs warnings reported for mapreduce-client-core while fixing MAPREDUCE-6223 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-4815: --- Attachment: MAPREDUCE-4815.v10.patch FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6227) DFSIO for truncate
Konstantin Shvachko created MAPREDUCE-6227: -- Summary: DFSIO for truncate Key: MAPREDUCE-6227 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6227 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, test Reporter: Konstantin Shvachko Create a benchmark and a test for truncate within the framework of TestDFSIO. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293009#comment-14293009 ] Siqi Li commented on MAPREDUCE-4815: Thanks for your feedback,[~jira.shegalov]. I have uploaded the patch to address the issues that you mentioned above. For the question of Why the flag for the new behavior is not initialized when FileOutputCommitter#FileOutputCommitter(Path, TaskAttemptContext) is used. I think it's calling this(outputPath, (JobContext)context), so the algorithmVersion will be initialized there FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Summary: MR job will fail after commit fail (was: MR job fails during commit fail) MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job fails during commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Description: when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. was:While testing RM HA, we ran into this issue where if the RM fails over while an MR AM is in the middle of a commit, the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. MR job fails during commit fail --- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291561#comment-14291561 ] Yang Hao commented on MAPREDUCE-5718: - if crashed during a commit and if an api comes out to check if the job can fail over, then the problem will be fixed MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reopened MAPREDUCE-5718: - MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5067) native taskcontroller won't build on Ubuntu 12.10
[
https://issues.apache.org/jira/browse/MAPREDUCE-5067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-5067:
--
Resolution: Cannot Reproduce
Fix Version/s: 2.7.0
Status: Resolved (was: Patch Available)
nobody has seen this for a while
native taskcontroller won't build on Ubuntu 12.10
-
Key: MAPREDUCE-5067
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5067
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Affects Versions: 1.3.0
Environment: Ubuntu 12.10 server 64 bit, openjdk7
Reporter: Steve Loughran
Priority: Minor
Fix For: 2.7.0
Attachments: MAPREDUCE-5067.patch
Building the native tarball is failing in {{task-controller.c}} with
{code}
[exec] /usr/include/fts.h:41:3: error: #error fts.h cannot be used with
-D_FILE_OFFSET_BITS==64
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[
https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292535#comment-14292535
]
Hadoop QA commented on MAPREDUCE-4815:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694627/MAPREDUCE-4815.v10.patch
against trunk revision 21d5599.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5122//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5122//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5122//console
This message is automatically generated.
FileOutputCommitter.commitJob can be very slow for jobs with many output files
--
Key: MAPREDUCE-4815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1
Reporter: Jason Lowe
Assignee: Siqi Li
Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v3.patch,
MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch,
MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch
If a job generates many files to commit then the commitJob method call at the
end of the job can take minutes. This is a performance regression from 1.x,
as 1.x had the tasks commit directly to the final output directory as they
were completing and commitJob had very little to do. The commit work was
processed in parallel and overlapped the processing of outstanding tasks. In
0.23/2.x, the commit is single-threaded and waits until all tasks have
completed before commencing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4348) JobSubmissionProtocol should be made public, not package private
[ https://issues.apache.org/jira/browse/MAPREDUCE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4348: -- Resolution: Won't Fix Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) JobSubmissionProtocol should be made public, not package private Key: MAPREDUCE-4348 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4348 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.0.3 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Fix For: 1.3.0 Attachments: MAPREDUCE-4348.patch Original Estimate: 0.5h Remaining Estimate: 0.5h The JobSubmissionProtocol interface is package private, yet it is the only way to remotely query the status of the JT or the cluster. Even if Job Submission is considered private, probing JT state shouldn't be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-512) Add tests for the DFS HTML and JSP pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-512. -- Resolution: Won't Fix Add tests for the DFS HTML and JSP pages Key: MAPREDUCE-512 URL: https://issues.apache.org/jira/browse/MAPREDUCE-512 Project: Hadoop Map/Reduce Issue Type: Test Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Original Estimate: 1h Remaining Estimate: 1h Add some basic tests to look for the standard JSP pages on a locally deployed MiniMR cluster 1. namenode: check that dfshealth is present 2. datanode: check that all the datanode JSPs load 3. GET the standard servlets. The initial checks can just use httpclient to GET the pages; no need (yet) for HtmlUnit. If the tests were designed to take optional URLs (e.g test.namenode.url and test.datanode.url) they could be run against processes brought up externally/remotely They would * help test that the JSP pages are being compiled down and bundled into the JARS * verify the classpath is getting set up right * check that the Jasper engine is working * check the servlets are all registering I've effectively had to do this in my own code; having a set of these tests inside hadoop would make it easier to point the blame at the classpath setup or something else. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-1413) Improve logging of progress/errors in the job and task trackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-1413. --- Resolution: Won't Fix Improve logging of progress/errors in the job and task trackers --- Key: MAPREDUCE-1413 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.203.0, 0.22.0 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Trivial Fix For: 0.22.1 Attachments: MAPREDUCE-1413.patch, MAPREDUCE-1413.patch I have code that improves the logging of the trackers as they start stop and fail, through # More logging of events # including exception strings and stacks when things go wrong People's whose JTs and TTs aren't behaving may appreciate this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6228) Add truncate operation to SLive
Konstantin Shvachko created MAPREDUCE-6228: -- Summary: Add truncate operation to SLive Key: MAPREDUCE-6228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6228 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, test Reporter: Konstantin Shvachko Add truncate into the mix of operations for SLive test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
