[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-18 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170416#comment-16170416
 ] 

Hudson commented on MAPREDUCE-6954:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12898 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12898/])
MAPREDUCE-6954. Disable erasure coding for files that are uploaded to (rkanter: 
rev 0adc0471d0c06f66a31060f270dcb50a7b4ffafa)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploader.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java


> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-18 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170389#comment-16170389
 ] 

Peter Bacsko commented on MAPREDUCE-6954:
-

Thanks for fixing it.

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-18 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170383#comment-16170383
 ] 

Robert Kanter commented on MAPREDUCE-6954:
--

Oops, I was committing some other stuff that day too, and accidentally put the 
name of one of those in the commit messages :(
trunk: 
https://github.com/apache/hadoop/commit/3a8d57a0a2e047b34be82f602a2b6cf5593d2125
branch-3.0: 
https://github.com/apache/hadoop/commit/293ab432e116b93ada6ee56e518ae07aa853ab41

I've just reverted those two:
trunk: 
https://github.com/apache/hadoop/commit/5f496683fb00ba26a6bf5a506ae87d4bc4088727
https://github.com/apache/hadoop/commit/3847b8c861205d9c85c866ec3477e8b0d19bd659

And recommitted the code with the correct message:
trunk: 
https://github.com/apache/hadoop/commit/0adc0471d0c06f66a31060f270dcb50a7b4ffafa
https://github.com/apache/hadoop/commit/917e875e511589c297971f6a14fd85de7925409f

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-18 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169882#comment-16169882
 ] 

Peter Bacsko commented on MAPREDUCE-6954:
-

[~rkanter] something is not quite right. I can't see this change on any of the 
branches you mentioned. 

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-15 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168352#comment-16168352
 ] 

Robert Kanter commented on MAPREDUCE-6954:
--

+1

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-15 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167776#comment-16167776
 ] 

Hadoop QA commented on MAPREDUCE-6954:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
44s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887325/MAPREDUCE-6954-004.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 8a7126aa2100 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 50764ef |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7138/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7138/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-15 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167751#comment-16167751
 ] 

Peter Bacsko commented on MAPREDUCE-6954:
-

I had to upload a new patch, because I forgot to modify mapred-default.xml

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-15 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167742#comment-16167742
 ] 

Hadoop QA commented on MAPREDUCE-6954:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
7s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887316/MAPREDUCE-6954-003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 6d897802dcb4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 50764ef |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7137/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7137/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-15 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167708#comment-16167708
 ] 

Peter Bacsko commented on MAPREDUCE-6954:
-

I changed the boolean parameter as you requested. I agree it was confusing.
The default value of the property is now "false" instead of "true", that is it 
will try to disable EC.

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, 
> MAPREDUCE-6954-003.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-14 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166857#comment-16166857
 ] 

Robert Kanter commented on MAPREDUCE-6954:
--

Almost there:
# I'm not a fan of double negatives in booleans - they make things hard to 
read.  I understand what you're doing here, relying on the default value to be 
true instead of setting it (that's a good idea btw), but instead of using 
{{!shouldBeDisabled}} let's use {{default}} instead.  That's much easier to 
follow.
#- In a similar vein, let's rename {{testErasureCodingNotDisabled}} to 
{{testErasureCodingDefault}}.
# There's some new checkstyle warnings about indenting and line length.

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166590#comment-16166590
 ] 

Hadoop QA commented on MAPREDUCE-6954:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
22s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: 
The patch generated 3 new + 538 unchanged - 0 fixed = 541 total (was 538) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
42s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 47s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887129/MAPREDUCE-6954-002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 236c63233a8f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / b9465bb |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7134/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7134/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7134/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Disable erasure coding for files that are uploaded to the MR staging area
>

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-11 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162283#comment-16162283
 ] 

Robert Kanter commented on MAPREDUCE-6954:
--

Overall looks good.  A couple minor things:
- Update mapred-default like you suggested
- The staging dir config is {{yarn.app.mapreduce.am.staging-dir}} and defined 
in MRJobConfig as {{MR_AM_STAGING_DIR = MR_AM_PREFIX+"staging-dir"}}.  I think 
we should be consistent with this and make 
{{MR_JOB_STAGING_ERASURECODING_ENABLED = 
"mapreduce.job.staging-dir.erasurecoding.enabled"}} be 
{{MR_AM_STAGING_DIR_ERASURECODING_ENABLED = MR_AM_STAGING_DIR + 
"erasurecoding.enabled"}} which would resolve to 
{{yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled}}.
- It would be good to add a unit test.

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158646#comment-16158646
 ] 

Hadoop QA commented on MAPREDUCE-6954:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: 
The patch generated 2 new + 530 unchanged - 0 fixed = 532 total (was 530) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
55s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12886071/MAPREDUCE-6954-001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 122d57defd22 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5bbca80 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7124/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7124/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7124/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |

[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area

2017-09-08 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158602#comment-16158602
 ] 

Peter Bacsko commented on MAPREDUCE-6954:
-

Forgot to update {{mapred-default.xml}}, will do it in the next patch.

> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might 
> negatively affect the performance of an MR job if too many files are 
> localized.
> In such a scenario, users might want to disable EC in the staging area to 
> speed up the execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org