[jira] [Commented] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592032#comment-13592032
]
Hadoop QA commented on MAPREDUCE-3685:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571850/MAPREDUCE-3685.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3379//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3379//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3379//console
This message is automatically generated.
> There are some bugs in implementation of MergeManager
> -
>
> Key: MAPREDUCE-3685
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: anty.rao
>Assignee: anty
>Priority: Critical
> Attachments: MAPREDUCE-3685-branch-0.23.1.patch,
> MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch,
> MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
> MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch,
> MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3685: Target Version/s: 2.0.3-alpha, 3.0.0, 0.23.7 (was: 3.0.0, 2.0.3-alpha, 0.23.7) Status: Patch Available (was: Open) > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Critical > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3685: Attachment: MAPREDUCE-3685.patch Updated patch for trunk > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Critical > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3685: Attachment: MAPREDUCE-3685.branch-0.23.patch Updated patch for branch-0.23 > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Critical > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3685: Target Version/s: 2.0.3-alpha, 3.0.0, 0.23.7 (was: 3.0.0, 2.0.3-alpha, 0.23.7) Status: Open (was: Patch Available) > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Critical > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[
https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591943#comment-13591943
]
Hadoop QA commented on MAPREDUCE-5027:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571838/MAPREDUCE-5027-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3378//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3378//console
This message is automatically generated.
> Shuffle does not limit number of outstanding connections
>
>
> Key: MAPREDUCE-5027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch,
> MAPREDUCE-5027.patch, MAPREDUCE-5027.patch
>
>
> The ShuffleHandler does not have any configurable limits to the number of
> outstanding connections allowed. Therefore a node with many map outputs and
> many reducers in the cluster trying to fetch those outputs can exhaust a
> nodemanager out of file descriptors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5027) Shuffle does not limit number of outstanding connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-5027: - Attachment: MAPREDUCE-5027-3.patch John, I modified the test to try to open numerous connections and then verify the max is not reached. Thus eliminating the need do check the connect count repeatedly. test-patch.sh passed locally as did mvn test -Dtest=TestShuffleHandler. > Shuffle does not limit number of outstanding connections > > > Key: MAPREDUCE-5027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5027 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Robert Parker > Attachments: MAPREDUCE-5027-2.patch, MAPREDUCE-5027-3.patch, > MAPREDUCE-5027.patch, MAPREDUCE-5027.patch > > > The ShuffleHandler does not have any configurable limits to the number of > outstanding connections allowed. Therefore a node with many map outputs and > many reducers in the cluster trying to fetch those outputs can exhaust a > nodemanager out of file descriptors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1176) Contribution: FixedLengthInputFormat and FixedLengthRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591931#comment-13591931
]
Jonathan Clark commented on MAPREDUCE-1176:
---
This patch appeared to be within inches of being incorporated 3 years ago and
this feature is still sorely missing from Hadoop. Where do things stand now?
> Contribution: FixedLengthInputFormat and FixedLengthRecordReader
>
>
> Key: MAPREDUCE-1176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1176
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 0.20.1, 0.20.2
> Environment: Any
>Reporter: BitsOfInfo
>Priority: Minor
> Attachments: MAPREDUCE-1176-v1.patch, MAPREDUCE-1176-v2.patch,
> MAPREDUCE-1176-v3.patch, MAPREDUCE-1176-v4.patch
>
>
> Hello,
> I would like to contribute the following two classes for incorporation into
> the mapreduce.lib.input package. These two classes can be used when you need
> to read data from files containing fixed length (fixed width) records. Such
> files have no CR/LF (or any combination thereof), no delimiters etc, but each
> record is a fixed length, and extra data is padded with spaces. The data is
> one gigantic line within a file.
> Provided are two classes first is the FixedLengthInputFormat and its
> corresponding FixedLengthRecordReader. When creating a job that specifies
> this input format, the job must have the
> "mapreduce.input.fixedlengthinputformat.record.length" property set as follows
> myJobConf.setInt("mapreduce.input.fixedlengthinputformat.record.length",[myFixedRecordLength]);
> OR
> myJobConf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH,
> [myFixedRecordLength]);
> This input format overrides computeSplitSize() in order to ensure that
> InputSplits do not contain any partial records since with fixed records there
> is no way to determine where a record begins if that were to occur. Each
> InputSplit passed to the FixedLengthRecordReader will start at the beginning
> of a record, and the last byte in the InputSplit will be the last byte of a
> record. The override of computeSplitSize() delegates to FileInputFormat's
> compute method, and then adjusts the returned split size by doing the
> following: (Math.floor(fileInputFormatsComputedSplitSize / fixedRecordLength)
> * fixedRecordLength)
> This suite of fixed length input format classes, does not support compressed
> files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
