[jira] [Commented] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650690#comment-16650690
]
Hadoop QA commented on MAPREDUCE-7150:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 4 new +
179 unchanged - 6 fixed = 183 total (was 185) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
46s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7150 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943986/YARN-8872.04.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 17797c72d8a4 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
|
[jira] [Updated] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Description:
Currently, {{JobSplitWriter}} compares the number of hosts for a certain block
against a static value that comes from {{mapreduce.job.max.split.locations}}.
The default value of this property is 10.
However, an EC schema like RS-10-4 requires at least 14 hosts. In this case, 14
block locations will be returned and {{JobSplitWriter}} prints a warning, which
can confuse users.
A possible solution could check whether EC is enabled for a block and increase
this value dynamically if needed. A simpler one is to simply increase the
default value to a sensible number like 15.
was:
Currently, {{JobSplitWriter}} compares the number of hosts for a certain block
against a static value that comes from {{mapreduce.job.max.split.locations}}.
The default value of this property is 10.
However, an EC shema like RS-10-4 requires at least 14 hosts. In this case, 14
block locations will be returned and {{JobSplitWriter}} prints a warning, which
can confuse users.
A possible solution could check whether EC is enabled for a block and increase
this value dynamically if needed. A simpler one is to simply increase the
default value to a sensible number like 15.
> JobSplitWriter prints unnecessary warnings if EC is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC schema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650647#comment-16650647
]
Peter Bacsko commented on MAPREDUCE-7132:
-
[~xiaochen] yes, that's right. I'll change that tomorrow.
> JobSplitWriter prints unnecessary warnings if EC is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Summary: JobSplitWriter prints unnecessary warnings if EC is used (was:
Check erasure coding in JobSplitWriter to avoid warnings)
> JobSplitWriter prints unnecessary warnings if EC is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated MAPREDUCE-7150:
--
Status: Patch Available (was: In Progress)
Fixed a couple of checkstyle warnings (the others are existing, and I
personally would prefer the code to look the way it looks now). Also updated
the git commit line. [~haibochen] assuming the tests pass, I think this patch
is good to integrate?
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated MAPREDUCE-7150:
--
Attachment: YARN-8872.04.patch
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated MAPREDUCE-7150:
--
Status: In Progress (was: Patch Available)
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650502#comment-16650502
]
Xiao Chen commented on MAPREDUCE-7132:
--
Thanks [~pbacsko] for revving. Should we also change MRConfig.java and update
{{MAX_BLOCK_LOCATIONS_DEFAULT}} to be consistent with the xml?
+1 pending
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650429#comment-16650429
]
Hadoop QA commented on MAPREDUCE-7152:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 5s{color} | {color:orange} root: The patch generated 8 new + 735 unchanged
- 0 fixed = 743 total (was 735) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
3s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 22s{color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.TestMapreduceConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7152 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943951/MAPREDUCE-7152-lazyEval_POC01.patc
[jira] [Updated] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7152:
Attachment: MAPREDUCE-7152-lazyEval_POC01.patch
> LD_LIBRARY_PATH is always passed from MR AM to tasks
>
>
> Key: MAPREDUCE-7152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7152-lazyEval_POC01.patch
>
>
> {{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default
> in Hadoop (as part of {{mapreduce.admin.user.env}} and
> {{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable
> from AM container to task containers in the container launch context.
> In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
> tasks will fail to load native library. A reliable way to fix this is to add
> {{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
> Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on
> the NM side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7152:
Status: Patch Available (was: Open)
> LD_LIBRARY_PATH is always passed from MR AM to tasks
>
>
> Key: MAPREDUCE-7152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7152-lazyEval_POC01.patch
>
>
> {{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default
> in Hadoop (as part of {{mapreduce.admin.user.env}} and
> {{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable
> from AM container to task containers in the container launch context.
> In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
> tasks will fail to load native library. A reliable way to fix this is to add
> {{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
> Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on
> the NM side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
Peter Bacsko created MAPREDUCE-7152:
---
Summary: LD_LIBRARY_PATH is always passed from MR AM to tasks
Key: MAPREDUCE-7152
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Peter Bacsko
Assignee: Peter Bacsko
{{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default in
Hadoop (as part of {{mapreduce.admin.user.env}} and
{{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable from
AM container to task containers in the container launch context.
In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
tasks will fail to load native library. A reliable way to fix this is to add
{{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on the
NM side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650103#comment-16650103
]
Hadoop QA commented on MAPREDUCE-7132:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
23s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943917/MAPREDUCE-7132-007.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux f5401154c4a3 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b4a38e7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7505/testReport/ |
| Max. process+thread count | 1645 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE
[jira] [Commented] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650058#comment-16650058
]
Peter Bacsko commented on MAPREDUCE-7132:
-
Patch v7: minor checkstyle fixes.
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Attachment: MAPREDUCE-7132-007.patch
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650057#comment-16650057
]
Hadoop QA commented on MAPREDUCE-7132:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
22s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943906/MAPREDUCE-7132-006.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux a05d3489d196 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5033deb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7504/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build
[jira] [Assigned] (MAPREDUCE-7151) RMContainerAllocator#handleJobPriorityChange expects application_priority always
[
https://issues.apache.org/jira/browse/MAPREDUCE-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T reassigned MAPREDUCE-7151:
Assignee: Bilwa S T
> RMContainerAllocator#handleJobPriorityChange expects application_priority
> always
>
>
> Key: MAPREDUCE-7151
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7151
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bilwa S T
>Priority: Major
>
> As per yarn_service.proto {{AllocateResponseProto}} *application_priority* is
> *optional* field.
> But {{RMContainerAllocator#handleJobPriorityChange}} expects to have
> application priority always in response which is not mandatory.
> {code}
> private void handleJobPriorityChange(AllocateResponse response) {
> Priority priorityFromResponse = Priority.newInstance(response
> .getApplicationPriority().getPriority());
> // Update the job priority to Job directly.
> getJob().setJobPriority(priorityFromResponse);
> }
> {code}
> {code}
> message AllocateResponseProto {
> ..
> optional hadoop.common.TokenProto am_rm_token = 12;
> optional PriorityProto application_priority = 13;
> optional CollectorInfoProto collector_info = 14;
> ..
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7151) RMContainerAllocator#handleJobPriorityChange expects application_priority always
Bibin A Chundatt created MAPREDUCE-7151:
---
Summary: RMContainerAllocator#handleJobPriorityChange expects
application_priority always
Key: MAPREDUCE-7151
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7151
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
As per yarn_service.proto {{AllocateResponseProto}} *application_priority* is
*optional* field.
But {{RMContainerAllocator#handleJobPriorityChange}} expects to have
application priority always in response which is not mandatory.
{code}
private void handleJobPriorityChange(AllocateResponse response) {
Priority priorityFromResponse = Priority.newInstance(response
.getApplicationPriority().getPriority());
// Update the job priority to Job directly.
getJob().setJobPriority(priorityFromResponse);
}
{code}
{code}
message AllocateResponseProto {
..
optional hadoop.common.TokenProto am_rm_token = 12;
optional PriorityProto application_priority = 13;
optional CollectorInfoProto collector_info = 14;
..
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Description:
Currently, {{JobSplitWriter}} compares the number of hosts for a certain block
against a static value that comes from {{mapreduce.job.max.split.locations}}.
The default value of this property is 10.
However, an EC shema like RS-10-4 requires at least 14 hosts. In this case, 14
block locations will be returned and {{JobSplitWriter}} prints a warning, which
can confuse users.
A possible solution could check whether EC is enabled for a block and increase
this value dynamically if needed. A simpler one is to simply increase the
default value to a sensible number like 15.
was:
Currently, {{JobSplitWriter}} compares the number of hosts for a certain block
against a static value that comes from {{mapreduce.job.max.split.locations}}.
The default value of this property is 10.
However, an EC shema like RS-10-4 requires at least 14 hosts. In this case, 14
block locations will be returned and {{JobSplitWriter}} prints a warning, which
can confuse users.
A possible solution could check whether EC is enabled for a block and increase
this value dynamically if needed. A simpler one is to simply increase the
default value to a sensible value like 15.
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Attachment: MAPREDUCE-7132-006.patch
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible value like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Description:
Currently, {{JobSplitWriter}} compares the number of hosts for a certain block
against a static value that comes from {{mapreduce.job.max.split.locations}}.
The default value of this property is 10.
However, an EC shema like RS-10-4 requires at least 14 hosts. In this case, 14
block locations will be returned and {{JobSplitWriter}} prints a warning, which
can confuse users.
A possible solution could check whether EC is enabled for a block and increase
this value dynamically if needed. A simpler one is to simply increase the
default value to a sensible value like 15.
was:
Currently, {{JobSplitWriter}} compares the number of hosts for a certain block
against a static value that comes from {{mapreduce.job.max.split.locations}}.
The default value of this property is 10.
However, an EC shema like RS-10-4 requires at least 14 hosts. In this case, 14
block locations will be returned and {{JobSplitWriter}} prints a warning, which
can confuse users.
A possible solution could check whether EC is enabled for a block and increase
this value dynamically if needed.
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible value like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) Check erasure coding in JobSplitWriter to avoid warnings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649962#comment-16649962
]
Peter Bacsko commented on MAPREDUCE-7132:
-
[~xiaochen] thanks, it makes sense. I think it is a rare scenario, but still,
it can happen.
I'll bump the default to 15 and update the tests a little bit.
> Check erasure coding in JobSplitWriter to avoid warnings
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC shema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
