[jira] [Commented] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225890#comment-13225890 ] Hadoop QA commented on MAPREDUCE-3991: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517680/MAPREDUCE-3991.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2023//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2023//console This message is automatically generated. > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
[
https://issues.apache.org/jira/browse/MAPREDUCE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225875#comment-13225875
]
Hadoop QA commented on MAPREDUCE-3990:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517679/MAPREDUCE-3990.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2022//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2022//console
This message is automatically generated.
> MRBench allows Long-sized input-lines value but parses CLI argument as an
> Integer
> -
>
> Key: MAPREDUCE-3990
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-3990.patch
>
>
> MRBench has the following method:
> {code}
> public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
> Order sortOrder) { ... }
> {code}
> The method is already set to accept a long datatype for numLines, for
> generating very large amount of data.
> However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via
> an Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
> NumberFormatExceptions as a result.
> The parsing should be Long.parseLong and the inputLines datatype should be
> switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Patch Available (was: Open) > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3991: --- Attachment: MAPREDUCE-3991.patch > Streaming FAQ has some wrong instructions about input files splitting > - > > Key: MAPREDUCE-3991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-3991.patch > > > Steaming docs say, at: > http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F > "Generate a file containing the full HDFS path of the input files. Each map > task would get one file name as input." > This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3991) Streaming FAQ has some wrong instructions about input files splitting
Streaming FAQ has some wrong instructions about input files splitting - Key: MAPREDUCE-3991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3991 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 0.23.0 Reporter: Harsh J Assignee: Harsh J Priority: Trivial Steaming docs say, at: http://hadoop.apache.org/common/docs/current/streaming.html#How+do+I+process+files%2C+one+per+map%3F "Generate a file containing the full HDFS path of the input files. Each map task would get one file name as input." This is incorrect, as a file isn't split by lines, rather by size - for MR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
[
https://issues.apache.org/jira/browse/MAPREDUCE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3990:
---
Target Version/s: 0.24.0, 0.23.3 (was: 0.24.0)
Status: Patch Available (was: Open)
> MRBench allows Long-sized input-lines value but parses CLI argument as an
> Integer
> -
>
> Key: MAPREDUCE-3990
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-3990.patch
>
>
> MRBench has the following method:
> {code}
> public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
> Order sortOrder) { ... }
> {code}
> The method is already set to accept a long datatype for numLines, for
> generating very large amount of data.
> However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via
> an Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
> NumberFormatExceptions as a result.
> The parsing should be Long.parseLong and the inputLines datatype should be
> switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
[
https://issues.apache.org/jira/browse/MAPREDUCE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3990:
---
Attachment: MAPREDUCE-3990.patch
> MRBench allows Long-sized input-lines value but parses CLI argument as an
> Integer
> -
>
> Key: MAPREDUCE-3990
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-3990.patch
>
>
> MRBench has the following method:
> {code}
> public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
> Order sortOrder) { ... }
> {code}
> The method is already set to accept a long datatype for numLines, for
> generating very large amount of data.
> However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via
> an Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
> NumberFormatExceptions as a result.
> The parsing should be Long.parseLong and the inputLines datatype should be
> switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
MRBench allows Long-sized input-lines value but parses CLI argument as an
Integer
-
Key: MAPREDUCE-3990
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.23.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: MAPREDUCE-3990.patch
MRBench has the following method:
{code}
public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
Order sortOrder) { ... }
{code}
The method is already set to accept a long datatype for numLines, for
generating very large amount of data.
However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via an
Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
NumberFormatExceptions as a result.
The parsing should be Long.parseLong and the inputLines datatype should be
switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225697#comment-13225697 ] Hadoop QA commented on MAPREDUCE-3989: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517635/MAPREDUCE-3989.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2021//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2021//console This message is automatically generated. > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3578) starting nodemanager as 'root' gives "Unknown -jvm option"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225682#comment-13225682
]
Alejandro Abdelnur commented on MAPREDUCE-3578:
---
+1
> starting nodemanager as 'root' gives "Unknown -jvm option"
> --
>
> Key: MAPREDUCE-3578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3578
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0
> Environment: jdk 1.6.0.26, ubuntu 11.04
>Reporter: Gilad Wolff
> Attachments: MAPREDUCE-3578.patch
>
>
> running "sudo HADOOP_ROOT/bin/yarn-daemon.sh start nodemanager" I get
> "unknown -jvm option" (jdk version 1.6.0.26). The problem seems to be with
> line 204 in yarn:
> elif [ "$COMMAND" = "nodemanager" ] ; then
> CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
> CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
> if [[ $EUID -eq 0 ]]; then
> YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS"
> else
> YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
> fi
> using -server seems to solve the problem for me.
> I tested using build 929 from
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3578) starting nodemanager as 'root' gives "Unknown -jvm option"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur reassigned MAPREDUCE-3578:
-
Assignee: Tom White
> starting nodemanager as 'root' gives "Unknown -jvm option"
> --
>
> Key: MAPREDUCE-3578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3578
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0
> Environment: jdk 1.6.0.26, ubuntu 11.04
>Reporter: Gilad Wolff
>Assignee: Tom White
> Attachments: MAPREDUCE-3578.patch
>
>
> running "sudo HADOOP_ROOT/bin/yarn-daemon.sh start nodemanager" I get
> "unknown -jvm option" (jdk version 1.6.0.26). The problem seems to be with
> line 204 in yarn:
> elif [ "$COMMAND" = "nodemanager" ] ; then
> CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
> CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
> if [[ $EUID -eq 0 ]]; then
> YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS"
> else
> YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
> fi
> using -server seems to solve the problem for me.
> I tested using build 929 from
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt reassigned MAPREDUCE-3989: --- Assignee: Patrick Hunt > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-3989: Status: Patch Available (was: Open) > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-3989: Attachment: MAPREDUCE-3989.patch Updated patch with mr only in this patch. I re-verified by running from tarball with DEBUG turned on and the maxfilesize set to just 10k. > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225642#comment-13225642 ] Hadoop QA commented on MAPREDUCE-3940: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517621/MAPREDUCE-3940-20120308.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2020//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2020//console This message is automatically generated. > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
cap space usage of default log4j rolling policy (mr specific changes) - Key: MAPREDUCE-3989 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Patrick Hunt see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3105) NM<->RM shared secrets should be rolled every so often.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3105. Resolution: Duplicate Fix Version/s: (was: 0.24.0) Duplicate of MAPREDUCE-3943. > NM<->RM shared secrets should be rolled every so often. > > > Key: MAPREDUCE-3105 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3105 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security >Reporter: Vinod Kumar Vavilapalli > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225584#comment-13225584 ] Hadoop QA commented on MAPREDUCE-1740: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517623/MAPREDUCE-1740_branch-1.0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2019//console This message is automatically generated. > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.3 >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, > mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-1740: Attachment: MAPREDUCE-1740_branch-1.0.patch Thanks Steve and Todd for the review! I am attaching an updated patch for branch-1.0 incorporating Steve's style comments. The patch mainly is to avoid the NPE that we are seeing with some users when the compared nodes are at different depth. > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.3 >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, MAPREDUCE-1740_branch-1.0.patch, > mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Status: Patch Available (was: Open) > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Attachment: MAPREDUCE-3940-20120308.txt Attaching patch that adds expiry time to container-token-ids and makes NM reject containers with expired tokens. Added a unit test to exactly validate the above. > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Status: Open (was: Patch Available) > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Attachment: MR-3796.3.patch Made min/max settings global at the RM level instead of the per-scheduler settings. > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Status: Patch Available (was: Open) > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3796) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3796: --- Hadoop Flags: Incompatible change > Scheduler.normalizeRequest does not account for allocation requests that > exceed maximumAllocation limits > - > > Key: MAPREDUCE-3796 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3796 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.2 > > Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, > MR-3796.wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't point to a single directory on a node.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne reassigned MAPREDUCE-3988: - Assignee: Eric Payne > mapreduce.job.local.dir doesn't point to a single directory on a node. > -- > > Key: MAPREDUCE-3988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Eric Payne > > After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks > but it doesn't point to the same directory for all tasks running on the node. > It is a public API. Either we should point to a single directory or point it > to all directories and change the documentation to say that it points to all > dirs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225434#comment-13225434 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3975: Created MAPREDUCE-3988. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't point to a single directory on a node.
mapreduce.job.local.dir doesn't point to a single directory on a node. -- Key: MAPREDUCE-3988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks but it doesn't point to the same directory for all tasks running on the node. It is a public API. Either we should point to a single directory or point it to all directories and change the documentation to say that it points to all dirs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3975. Resolution: Fixed Release Note: Exporting mapreduce.job.local.dir for mapreduce tasks to use as job-level shared scratch space. Hadoop Flags: Reviewed > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225428#comment-13225428 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3975: bq. There are no guarantees that the tasks will run on the same node, so any mapreduce jobs that use this need to be able to handle a task running by itself. The feature helps to share resources on a node, whatever tasks run on this node can access this, more like the private distributed cache. bq. can we open a new JIRA for this? Sure, let me close this and open a new ticket. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3982: --- Status: Open (was: Patch Available) Patch looks good. Can you please port the test TestEmptyJob to maven build and possibly MiniMRYarnCluster? Tx. > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225412#comment-13225412 ] Eric Payne commented on MAPREDUCE-3975: --- s/Arun/Vinod/g Sorry about that. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225401#comment-13225401 ] Robert Joseph Evans commented on MAPREDUCE-3975: Vinod, can we open a new JIRA for this? The patch has already been merged in, and it is a step closer to what we want, although obviously not all the way there. If the directories are not the same the worst thing that happens is that some resources are wasted. There are no guarantees that the tasks will run on the same node, so any mapreduce jobs that use this need to be able to handle a task running by itself. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225390#comment-13225390 ] Eric Payne commented on MAPREDUCE-3975: --- Thanks Arun. Can you please give me pointers where this config property should be set in order to ensure that it is the same for different task attempts on the same node? > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened MAPREDUCE-3975: Sorry for coming late on this: bq. I think that it is fine to have them in different base directories, if it becomes a problem in practice because tasks are trying to share data with each other, then it can be fixed then. This is exactly what the job.local.dir is intended for - sharing of data across tasks on the node. If the tasks only need scratch space, they can write in ./tmp directory which is already created by nodemanager. Even our documentation for the 20.**/1.** release lines explicitly says this. We need to fix it. I am reopening this ticket. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3975-1.txt > > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3978) getTotalMaps, getTotalReduces is incorrect for JobHistory PartialJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225355#comment-13225355 ] Siddharth Seth commented on MAPREDUCE-3978: --- Removing them is one option. Was thinking of just adding this info to the history file name. It's useful to have when a job fails - but redundant info for successful jobs. > getTotalMaps, getTotalReduces is incorrect for JobHistory PartialJob > > > Key: MAPREDUCE-3978 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3978 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth > > These are currently returning the number of completed map / reduce tasks - > instead of the total. The information doesn't make it over to the history > server via the job history file name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3729) Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225349#comment-13225349 ] Robert Joseph Evans commented on MAPREDUCE-3729: Oh and all of the failures appear to be on hadoop3, so my guess is that it is something with the box. But once I know what the problem is we might need to make some changes so the box does not get into that state again. > Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, > TestLocalModeWithNewApis > -- > > Key: MAPREDUCE-3729 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3729 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.1 > > > See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3729) Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225346#comment-13225346 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3729: I ran into this sometime back on my machine too. This happens when the tests are run by some user but the temp directories are created and some other user tries to run the tests. I asked [~gkesavan] to cleanup the temp dirs on apache jenkins for now. We can fix our code to avoid this from happening. > Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, > TestLocalModeWithNewApis > -- > > Key: MAPREDUCE-3729 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3729 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.1 > > > See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3729) Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225334#comment-13225334 ] Robert Joseph Evans commented on MAPREDUCE-3729: Looking at the code there appear to be a couple of things happening. The code to create the submitJobDir calls FileSystem.mkdirs, which calls fs.mkdirs, and then tries to call fs.setPermission. If the first mkdirs fails the result is ignored and the chmod call throws the exception that we are seeing. There are lots of reasons why the mkdirs could fail. My guess is that either /tmp is full or /tmp/hadoop-jenkins/mapred/staging has over 32000 entires in it, which is the limit for ext3. Why is that my guess, well because when I looked at my directory I have close to 4000 in mine, and I have not been running nearly as long or under as much of a load as this one has. I need to log onto the box to be sure of any of that though. > Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, > TestLocalModeWithNewApis > -- > > Key: MAPREDUCE-3729 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3729 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.1 > > > See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225294#comment-13225294 ] Robert Joseph Evans commented on MAPREDUCE-3982: That would not really work. The directory structure is the following. /_temporary//... The problem was that the code assumed that /_temporary// was being created by the individual map/reduce tasks. If it was not then other code would blowup, so instead of just creating /_temporary in the setup from before we are now creating /_temporary// When we clean up from a job we still want to completely remove the _temporary directory, not just /_temporary// or there will be an extra directory sitting around in the directory that will cause other problems. > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3987) Reduce RM app memory footprint once app has completed
Reduce RM app memory footprint once app has completed
-
Key: MAPREDUCE-3987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3987
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Jason Lowe
The ResourceManager holds onto a configurable number of completed applications
(yarn.resource.max-completed-applications, defaults to 1), and the memory
footprint of these completed applications can be significant. For example, the
{{submissionContext}} in RMAppImpl contains references to protocolbuffer
objects and other items that probably aren't necessary to keep around once the
application has completed. We could significantly reduce the memory footprint
of the RM by releasing objects that are no longer necessary once an application
completes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225238#comment-13225238 ] Ravi Prakash commented on MAPREDUCE-3982: - Would we want to change this in cleanupJob(JobContext context) too? Would that make the getPendingJobAttemptsPath() methods useless? > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225231#comment-13225231 ] Robert Joseph Evans commented on MAPREDUCE-3971: Yes you are correct about the implementation of the /mapreduce/jobs API up until MAPREDUCE-3944. MAPREDUCE-3944 changed it so that /mapreduce/jobs will only return the equivalent of a PartialJob, not a CompletedJob. This reduces the load a lot, and is hopefully a temporary measure until more issues can be addressed as part of MAPREDUCE-3973. I am not sure of the best way to implement this, because it is something that we want to be able to do consistently across APIs if need be. But, it will probably be something like keeping the limit that is already in place and adding in a start parameter as well. We will probably also need to add in fields to the returned list indicating the start index and end index of the items returned, and possibly something else indicating the total number of objects that can be returned. The maximum limit would be in place regardless of what the limit is set in the URL. The real difficultly with this is that we now need to be able to maintain a guaranteed consistent ordering of the values returned or the start will be useless. > Job History web services need to have limits on the number of itmes they can > return. > > > Key: MAPREDUCE-3971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > > The Job History web services canput a very large load on the job history > server. We should put in a limit on the number of entries that can be > returned by the web service, and also add in the ability to modify the > starting location in the list, so that all entries can still be downlaoded. > Just not all at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3982) TestEmptyJob fails with FileNotFound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225222#comment-13225222 ] Robert Joseph Evans commented on MAPREDUCE-3982: The test failures are known and covered by MAPREDUCE:3729 There are no tests added, because it is fixing a test that was failing. > TestEmptyJob fails with FileNotFound > > > Key: MAPREDUCE-3982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Attachments: MR-3982.txt > > > TestEmptyJob fails because teh FileOutputCommitter expects a directory to be > created that is not created. The FileOutputCommitter should either ignore > the error or create the directory itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225205#comment-13225205
]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3971:
---
Hi Robert,
As per my understanding, */mapreduce/jobs* API causes a very large load on the
job history server. However, a user can mention limit also, ofcourse it can be
optional.
Intention of this ticket is to return only a limited number of items,
something like recent 50 or 100 among 2, in the absense of limit factor or
irrespective of limit factor?
I think, suppose say a user mentions limit as 50, even then all the jobs from
*intermediateListCache* & *jobListCache* are being fetched from JHS and again
iterating over the list in the HsWebServices, to return only required number of
jobs. If so, I think, we can move the limit factor to JHS to parse only that
many jobs, some thing like following API will be useful
{noformat}
public Map getNJobs(String queue, String user, long sTime, long
sEnd, long fStart, long fEnd, int n) {}
{noformat}
Please provide your comments.
> Job History web services need to have limits on the number of itmes they can
> return.
>
>
> Key: MAPREDUCE-3971
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.2
>Reporter: Robert Joseph Evans
>
> The Job History web services canput a very large load on the job history
> server. We should put in a limit on the number of entries that can be
> returned by the web service, and also add in the ability to modify the
> starting location in the list, so that all entries can still be downlaoded.
> Just not all at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
