[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220817#comment-13220817 ] Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3952: --- I *think*, input file might be an empty file. If so, CombineFileInputFormat#getSplits just returns empty *splits* list. Where as FileInputFormat#getSplits creats a file split of length 0 for each empty file and adds it to *splits* list. So the number of file splits is atleast 1. BTW Why do we need to submit a job having an empty input file(s) to framework? In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. --- Key: MAPREDUCE-3952 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Zhenxiao Luo Hive get unexpected result when using MR2(When using MR1, always get expected result). In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. The calling code in Hive, in Hadoop23Shims.java: InputSplit[] splits = super.getSplits(job, numSplits); this get splits.length == 0. In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, numSplits); this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3956) Remove the use of the deprecated Syncable.sync() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220895#comment-13220895 ] Hudson commented on MAPREDUCE-3956: --- Integrated in Hadoop-Hdfs-trunk #972 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/972/]) HADOOP-8124, HDFS-3034, MAPREDUCE-3956. Remove the deprecated Syncable.sync(). (Revision 1295999) Result = UNSTABLE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1295999 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Syncable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java Remove the use of the deprecated Syncable.sync() method --- Key: MAPREDUCE-3956 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3956 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Fix For: 0.24.0 Attachments: m3956_20120301.patch This is a part of HADOOP-8124. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3497) missing documentation for yarn cli and subcommands - similar to commands_manual.html
[ https://issues.apache.org/jira/browse/MAPREDUCE-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3497: - Attachment: MAPREDUCE-3497.patch MAPREDUCE-3497_site.patch split hadoop-project/src/site/site.xml into separate patch to appease jenkins. missing documentation for yarn cli and subcommands - similar to commands_manual.html Key: MAPREDUCE-3497 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3497 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3497.patch, MAPREDUCE-3497.patch, MAPREDUCE-3497_site.patch the yarn cli and sub-commands aren't documented anywhere. Should have documentation similar to the commands_manual.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3792) job -list displays only the jobs submitted by a particular user
[ https://issues.apache.org/jira/browse/MAPREDUCE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3792: -- Status: Open (was: Patch Available) Per discussion with Vinod, canceling patch to have mapred job -list translate resource amounts of -1 to N/A so the output looks a bit nicer. job -list displays only the jobs submitted by a particular user --- Key: MAPREDUCE-3792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3792 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch mapred job -list lists only the jobs submitted by the user who ran the command. This behavior is different from 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3956) Remove the use of the deprecated Syncable.sync() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220958#comment-13220958 ] Hudson commented on MAPREDUCE-3956: --- Integrated in Hadoop-Mapreduce-trunk #1007 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1007/]) HADOOP-8124, HDFS-3034, MAPREDUCE-3956. Remove the deprecated Syncable.sync(). (Revision 1295999) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1295999 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Syncable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java Remove the use of the deprecated Syncable.sync() method --- Key: MAPREDUCE-3956 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3956 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Fix For: 0.24.0 Attachments: m3956_20120301.patch This is a part of HADOOP-8124. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3850) Avoid redundant calls for tokens in TokenCache
[
https://issues.apache.org/jira/browse/MAPREDUCE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-3850:
---
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Status: Open (was: Patch Available)
Avoid redundant calls for tokens in TokenCache
--
Key: MAPREDUCE-3850
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3850
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: security
Affects Versions: 0.23.1, 0.24.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: MAPREDUCE-3850.patch, MAPREDUCE-3850.patch
The {{TokenCache}} will repeatedly call the same filesystem for tokens. This
is inefficient and can easily be changed to only call each filesystem once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3034: -- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) Upmerging to latest 0.23.2 branch. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3034: -- Attachment: MAPREDUCE-3034-5.patch I upmerged to the latest 0.23.2 branch. I ran several before and after tests to be sure that job times did not increase nor did heap usage increase for NM or client when comparing before RM stop/start vs. after RM stop/start. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3034: -- Target Version/s: 0.23.0, 0.24.0, 0.23.2 (was: 0.24.0, 0.23.0) Affects Version/s: 0.23.2 Status: Patch Available (was: Open) The MAPREDUCE-3034-5.patch applies to trunk as well. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0, 0.23.2 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3497) missing documentation for yarn cli and subcommands - similar to commands_manual.html
[ https://issues.apache.org/jira/browse/MAPREDUCE-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220993#comment-13220993 ] Hadoop QA commented on MAPREDUCE-3497: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516836/MAPREDUCE-3497.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1983//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1983//console This message is automatically generated. missing documentation for yarn cli and subcommands - similar to commands_manual.html Key: MAPREDUCE-3497 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3497 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3497.patch, MAPREDUCE-3497.patch, MAPREDUCE-3497_site.patch the yarn cli and sub-commands aren't documented anywhere. Should have documentation similar to the commands_manual.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3792) job -list displays only the jobs submitted by a particular user
[ https://issues.apache.org/jira/browse/MAPREDUCE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3792: -- Attachment: MAPREDUCE-3792.patch Updated patch to translate negative resource usage into N/A for mapred job -list output. job -list displays only the jobs submitted by a particular user --- Key: MAPREDUCE-3792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3792 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch mapred job -list lists only the jobs submitted by the user who ran the command. This behavior is different from 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3792) job -list displays only the jobs submitted by a particular user
[ https://issues.apache.org/jira/browse/MAPREDUCE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3792: -- Status: Patch Available (was: Open) job -list displays only the jobs submitted by a particular user --- Key: MAPREDUCE-3792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3792 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch mapred job -list lists only the jobs submitted by the user who ran the command. This behavior is different from 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221033#comment-13221033 ] Hadoop QA commented on MAPREDUCE-3034: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516844/MAPREDUCE-3034-5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1984//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1984//console This message is automatically generated. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0, 0.23.2 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221053#comment-13221053
]
Alejandro Abdelnur commented on MAPREDUCE-3837:
---
Mayank,
* Built branch-1 with your patch
* Configured the cluster, run a job test is OK
* Configured the mapred-site.xml with 'mapred.jobtracker.restart.recover=true'
* Restarted the JT
* Created a IN data file in my HDFS home dir
* Submitted 5 wordcount jobs
{code}
bin/hadoop jar hadoop-*examples*jar wordcount IN OUT0
bin/hadoop jar hadoop-*examples*jar wordcount IN OUT1
bin/hadoop jar hadoop-*examples*jar wordcount IN OUT2
bin/hadoop jar hadoop-*examples*jar wordcount IN OUT3
bin/hadoop jar hadoop-*examples*jar wordcount IN OUT4
{code}
* Waited till they are all running
* Killed the JT
* Restarted the JT
The jobs are not recovered, and what I see in the logs is:
{code}
2012-03-02 08:55:22,164 INFO org.apache.hadoop.mapred.JobTracker: Found an
incomplete job directory job_201203020852_0001. Deleting it!!
2012-03-02 08:55:22,194 INFO org.apache.hadoop.mapred.JobTracker: Found an
incomplete job directory job_201203020852_0002. Deleting it!!
2012-03-02 08:55:22,204 INFO org.apache.hadoop.mapred.JobTracker: Found an
incomplete job directory job_201203020852_0003. Deleting it!!
2012-03-02 08:55:22,224 INFO org.apache.hadoop.mapred.JobTracker: Found an
incomplete job directory job_201203020852_0004. Deleting it!!
2012-03-02 08:55:22,236 INFO org.apache.hadoop.mapred.JobTracker: Found an
incomplete job directory job_201203020852_0005. Deleting it!!
{code}
Am I missing some additional configuration?
Hadoop 22 Job tracker is not able to recover job in case of crash and after
that no user can submit job.
Key: MAPREDUCE-3837
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Fix For: 0.24.0, 0.22.1, 0.23.2
Attachments: PATCH-HADOOP-1-MAPREDUCE-3837.patch,
PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch
If job tracker is crashed while running , and there were some jobs are
running , so if job tracker's property mapreduce.jobtracker.restart.recover
is true then it should recover the job.
However the current behavior is as follows
jobtracker try to restore the jobs but it can not . And after that jobtracker
closes its handle to hdfs and nobody else can submit job.
Thanks,
Mayank
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3792) job -list displays only the jobs submitted by a particular user
[ https://issues.apache.org/jira/browse/MAPREDUCE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221061#comment-13221061 ] Hadoop QA commented on MAPREDUCE-3792: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516850/MAPREDUCE-3792.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1985//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1985//console This message is automatically generated. job -list displays only the jobs submitted by a particular user --- Key: MAPREDUCE-3792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3792 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch, MAPREDUCE-3792.patch mapred job -list lists only the jobs submitted by the user who ran the command. This behavior is different from 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221118#comment-13221118 ] Amol Kekre commented on MAPREDUCE-3896: --- pls commit asap as this is a blocker for other components of Hadoop stack pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221146#comment-13221146 ] Arun C Murthy commented on MAPREDUCE-3837: -- -1 on committing to branch-1. We've had innumerable issues with this before, not a good idea for a stable branch. Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job. Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3897) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3897: -- Attachment: MAPREDUCE-3897-2.txt Updated patch so that maxActiveApplications remains calculated based on absoluteMaxCapacity but modified so that only maxActiveApplicationsPerUser is calculated based on absoluteCapacity. This will limit the number of active apps for a single user to a percetage corresponding the absolute capacity of the queue unless the user limit factor is increased for that queue. capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong -- Key: MAPREDUCE-3897 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3897 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3897-1.txt, MAPREDUCE-3897-1.txt, MAPREDUCE-3897-2.txt The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser based on the config yarn.scheduler.capacity.maximum-applications or default 1. MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity), 1) MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100) * userLimitFactor), 1) maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so if max capacity capacity and if you have user limit factor 1 (which is the default) and only 1 user is running, that user will not be allowed to use over the queue capacity, so having it relative to MAX capacity doesn't make sense. That user could easily end up in a deadlock and all its space used by application masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3897) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3897: -- Status: Patch Available (was: Open) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong -- Key: MAPREDUCE-3897 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3897 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3897-1.txt, MAPREDUCE-3897-1.txt, MAPREDUCE-3897-2.txt The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser based on the config yarn.scheduler.capacity.maximum-applications or default 1. MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity), 1) MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100) * userLimitFactor), 1) maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so if max capacity capacity and if you have user limit factor 1 (which is the default) and only 1 user is running, that user will not be allowed to use over the queue capacity, so having it relative to MAX capacity doesn't make sense. That user could easily end up in a deadlock and all its space used by application masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3960) web proxy doesn't forward request to AM with configured hostname/IP
web proxy doesn't forward request to AM with configured hostname/IP --- Key: MAPREDUCE-3960 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3960 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical If the host the web proxy is running on has an ip alias or similar and the config file is pointing to the hostname that is the aliased ip of the box, the web proxy will send the request from the base ip rather then the aliased ip and the AM will redirect that request to the proxy again instead of accepting it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3960) web proxy doesn't forward request to AM with configured hostname/IP
[ https://issues.apache.org/jira/browse/MAPREDUCE-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3960: - Attachment: MAPREDUCE-3960.patch web proxy doesn't forward request to AM with configured hostname/IP --- Key: MAPREDUCE-3960 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3960 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3960.patch If the host the web proxy is running on has an ip alias or similar and the config file is pointing to the hostname that is the aliased ip of the box, the web proxy will send the request from the base ip rather then the aliased ip and the AM will redirect that request to the proxy again instead of accepting it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3960) web proxy doesn't forward request to AM with configured hostname/IP
[ https://issues.apache.org/jira/browse/MAPREDUCE-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221182#comment-13221182 ] Robert Joseph Evans commented on MAPREDUCE-3960: I think the change looks good. Did you run any tests on the code? There are almost no unit tests for the proxy code so manual testing is needed. web proxy doesn't forward request to AM with configured hostname/IP --- Key: MAPREDUCE-3960 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3960 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3960.patch If the host the web proxy is running on has an ip alias or similar and the config file is pointing to the hostname that is the aliased ip of the box, the web proxy will send the request from the base ip rather then the aliased ip and the AM will redirect that request to the proxy again instead of accepting it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221188#comment-13221188 ] Siddharth Seth commented on MAPREDUCE-3896: --- +1. Patch looks good. pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3896: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-0.23 and branch-0.23.2 Thanks Vinod for the fix, and John for verifying it. pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3932) MR tasks failing and crashing the AM when available-resources/headRoom becomes zero
[
https://issues.apache.org/jira/browse/MAPREDUCE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amol Kekre updated MAPREDUCE-3932:
--
Priority: Critical (was: Major)
Making it critical
MR tasks failing and crashing the AM when available-resources/headRoom
becomes zero
---
Key: MAPREDUCE-3932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3932
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Priority: Critical
Fix For: 0.23.2
[~karams] reported this offline. One reduce task gets preempted because of
zero headRoom and crashes the AM.
{code}
2012-02-23 11:30:15,956 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 AssignedMaps:0
AssignedReduces:0 completedMaps:4 completedReduces:0 containersAllocated:4
containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4
availableResources(headroom):memory: 44544
2012-02-23 11:30:16,959 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
Scheduling: PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23
AssignedMaps:0 AssignedReduces:0 completedMaps:4 completedReduces:0
containersAllocated:4 containersReleased:0 hostLocalAssigned:0
rackLocalAssigned:4 availableResources(headroom):memory: 44544
2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 AssignedMaps:0
AssignedReduces:0 completedMaps:4 completedReduces:0 containersAllocated:4
containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4
availableResources(headroom):memory: 0
2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Assign:
PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 AssignedMaps:0
AssignedReduces:0 completedMaps:4 completedReduces:0 containersAllocated:4
containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4
availableResources(headroom):memory: 0
2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
containers 3
2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
container container_1329995034628_0983_01_06 to
attempt_1329995034628_0983_r_00_0
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
container container_1329995034628_0983_01_07 to
attempt_1329995034628_0983_r_01_0
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
container container_1329995034628_0983_01_08 to
attempt_1329995034628_0983_r_02_0
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Assign:
PendingReduces:377 ScheduledMaps:6 ScheduledReduces:20 AssignedMaps:0
AssignedReduces:3 completedMaps:4 completedReduces:0 containersAllocated:7
containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4
availableResources(headroom):memory: 0
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all
scheduled reduces:20
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 2
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
attempt_1329995034628_0983_r_02_0
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
attempt_1329995034628_0983_r_01_0
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
schedule...
2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator]
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221196#comment-13221196 ] Hudson commented on MAPREDUCE-3896: --- Integrated in Hadoop-Hdfs-trunk-Commit #1896 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1896/]) MAPREDUCE-3896. Add user information to the delegation token issued by the history server. (Contributed by Vinod Kumar Vavilapalli) (Revision 1296401) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296401 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/krb5.conf pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221197#comment-13221197 ] Hudson commented on MAPREDUCE-3896: --- Integrated in Hadoop-Hdfs-0.23-Commit #619 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/619/]) merge MAPREDUCE-3896 from trunk (Revision 1296402) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/krb5.conf pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221200#comment-13221200 ] Hudson commented on MAPREDUCE-3896: --- Integrated in Hadoop-Common-trunk-Commit #1822 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1822/]) MAPREDUCE-3896. Add user information to the delegation token issued by the history server. (Contributed by Vinod Kumar Vavilapalli) (Revision 1296401) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296401 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/krb5.conf pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221204#comment-13221204 ] Hudson commented on MAPREDUCE-3896: --- Integrated in Hadoop-Common-0.23-Commit #630 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/630/]) merge MAPREDUCE-3896 from trunk (Revision 1296402) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/krb5.conf pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3897) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221226#comment-13221226 ] Hadoop QA commented on MAPREDUCE-3897: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516874/MAPREDUCE-3897-2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1986//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1986//console This message is automatically generated. capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong -- Key: MAPREDUCE-3897 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3897 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3897-1.txt, MAPREDUCE-3897-1.txt, MAPREDUCE-3897-2.txt The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser based on the config yarn.scheduler.capacity.maximum-applications or default 1. MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity), 1) MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100) * userLimitFactor), 1) maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so if max capacity capacity and if you have user limit factor 1 (which is the default) and only 1 user is running, that user will not be allowed to use over the queue capacity, so having it relative to MAX capacity doesn't make sense. That user could easily end up in a deadlock and all its space used by application masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3960) web proxy doesn't forward request to AM with configured hostname/IP
[ https://issues.apache.org/jira/browse/MAPREDUCE-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221227#comment-13221227 ] Thomas Graves commented on MAPREDUCE-3960: -- Yes I tested it manually, since as you said there aren't many unit tests and its pretty hard to unit test. created host with an IP alias and specified that as the config for the RM and verified requests failed without the change and then that requests succeeded with the change. web proxy doesn't forward request to AM with configured hostname/IP --- Key: MAPREDUCE-3960 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3960 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3960.patch If the host the web proxy is running on has an ip alias or similar and the config file is pointing to the hostname that is the aliased ip of the box, the web proxy will send the request from the base ip rather then the aliased ip and the AM will redirect that request to the proxy again instead of accepting it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3960) web proxy doesn't forward request to AM with configured hostname/IP
[ https://issues.apache.org/jira/browse/MAPREDUCE-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3960: - Status: Patch Available (was: Open) web proxy doesn't forward request to AM with configured hostname/IP --- Key: MAPREDUCE-3960 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3960 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3960.patch If the host the web proxy is running on has an ip alias or similar and the config file is pointing to the hostname that is the aliased ip of the box, the web proxy will send the request from the base ip rather then the aliased ip and the AM will redirect that request to the proxy again instead of accepting it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3497) missing documentation for yarn cli and subcommands - similar to commands_manual.html
[ https://issues.apache.org/jira/browse/MAPREDUCE-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3497: - Release Note: both MAPREDUCE-3497_site.patch and MAPREDUCE-3497.patch should be reviewed and committed since I split it apart to make jenkins happy. both MAPREDUCE-3497_site.patch and MAPREDUCE-3497.patch should be reviewed and committed since I split it apart to make jenkins happy. missing documentation for yarn cli and subcommands - similar to commands_manual.html Key: MAPREDUCE-3497 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3497 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3497.patch, MAPREDUCE-3497.patch, MAPREDUCE-3497_site.patch the yarn cli and sub-commands aren't documented anywhere. Should have documentation similar to the commands_manual.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3876) vertica query, sql command not properly ended
[ https://issues.apache.org/jira/browse/MAPREDUCE-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Doss updated MAPREDUCE-3876: --- Labels: hadoop newbie patch (was: ) Target Version/s: 1.0.0 Release Note: Apply to fresh version of 1.0.0 Status: Patch Available (was: Open) Only tested with 1.0.0 vertica query, sql command not properly ended - Key: MAPREDUCE-3876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3876 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.0 Environment: Red Hat 5.5 Oracle 11 Reporter: Joseph Doss Labels: newbie, patch, hadoop When running a test script, we're getting a java IO exception thrown. This test works on hadoop-0.20.0 but not on hadoop-1.0.0. Fri Feb 17 11:36:40 EST 2012 Running processes with name syncGL.sh: 0 LIB_JARS: /home/hadoop/verticasync/lib/vertica_4.1.14_jdk_5.jar,/home/hadoop/verticasync/lib/mail.jar,/home/hadoop/verticasync/lib/jdbc14.jar VERTICA_SYNC_JAR: /home/hadoop/verticasync/lib/vertica-sync.jar PROPERTIES_FILE: /home/hadoop/verticasync/config/ssp-vertica-sync-gl.properties Starting Vertica data sync - GL - process Warning: $HADOOP_HOME is deprecated. 12/02/17 11:36:43 INFO mapred.JobClient: Running job: job_201202171122_0001 12/02/17 11:36:44 INFO mapred.JobClient: map 0% reduce 0% 12/02/17 11:36:56 INFO mapred.JobClient: Task Id : attempt_201202171122_0001_m_00_0, Status : FAILED java.io.IOException: ORA-00933: SQL command not properly ended at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:289) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) 12/02/17 11:36:57 INFO mapred.JobClient: Task Id : attempt_201202171122_0001_m_01_0, Status : FAILED java.io.IOException: ORA-00933: SQL command not properly ended at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:289) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3876) vertica query, sql command not properly ended
[ https://issues.apache.org/jira/browse/MAPREDUCE-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Doss updated MAPREDUCE-3876: --- Attachment: HADOOP-oracleDriver-src.patch Only tested on a fresh version of 1.0.0 vertica query, sql command not properly ended - Key: MAPREDUCE-3876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3876 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.0 Environment: Red Hat 5.5 Oracle 11 Reporter: Joseph Doss Labels: hadoop, newbie, patch Attachments: HADOOP-oracleDriver-src.patch When running a test script, we're getting a java IO exception thrown. This test works on hadoop-0.20.0 but not on hadoop-1.0.0. Fri Feb 17 11:36:40 EST 2012 Running processes with name syncGL.sh: 0 LIB_JARS: /home/hadoop/verticasync/lib/vertica_4.1.14_jdk_5.jar,/home/hadoop/verticasync/lib/mail.jar,/home/hadoop/verticasync/lib/jdbc14.jar VERTICA_SYNC_JAR: /home/hadoop/verticasync/lib/vertica-sync.jar PROPERTIES_FILE: /home/hadoop/verticasync/config/ssp-vertica-sync-gl.properties Starting Vertica data sync - GL - process Warning: $HADOOP_HOME is deprecated. 12/02/17 11:36:43 INFO mapred.JobClient: Running job: job_201202171122_0001 12/02/17 11:36:44 INFO mapred.JobClient: map 0% reduce 0% 12/02/17 11:36:56 INFO mapred.JobClient: Task Id : attempt_201202171122_0001_m_00_0, Status : FAILED java.io.IOException: ORA-00933: SQL command not properly ended at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:289) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) 12/02/17 11:36:57 INFO mapred.JobClient: Task Id : attempt_201202171122_0001_m_01_0, Status : FAILED java.io.IOException: ORA-00933: SQL command not properly ended at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:289) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3876) vertica query, sql command not properly ended
[ https://issues.apache.org/jira/browse/MAPREDUCE-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221242#comment-13221242 ] Hadoop QA commented on MAPREDUCE-3876: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516880/HADOOP-oracleDriver-src.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1988//console This message is automatically generated. vertica query, sql command not properly ended - Key: MAPREDUCE-3876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3876 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.0 Environment: Red Hat 5.5 Oracle 11 Reporter: Joseph Doss Labels: hadoop, newbie, patch Attachments: HADOOP-oracleDriver-src.patch When running a test script, we're getting a java IO exception thrown. This test works on hadoop-0.20.0 but not on hadoop-1.0.0. Fri Feb 17 11:36:40 EST 2012 Running processes with name syncGL.sh: 0 LIB_JARS: /home/hadoop/verticasync/lib/vertica_4.1.14_jdk_5.jar,/home/hadoop/verticasync/lib/mail.jar,/home/hadoop/verticasync/lib/jdbc14.jar VERTICA_SYNC_JAR: /home/hadoop/verticasync/lib/vertica-sync.jar PROPERTIES_FILE: /home/hadoop/verticasync/config/ssp-vertica-sync-gl.properties Starting Vertica data sync - GL - process Warning: $HADOOP_HOME is deprecated. 12/02/17 11:36:43 INFO mapred.JobClient: Running job: job_201202171122_0001 12/02/17 11:36:44 INFO mapred.JobClient: map 0% reduce 0% 12/02/17 11:36:56 INFO mapred.JobClient: Task Id : attempt_201202171122_0001_m_00_0, Status : FAILED java.io.IOException: ORA-00933: SQL command not properly ended at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:289) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) 12/02/17 11:36:57 INFO mapred.JobClient: Task Id : attempt_201202171122_0001_m_01_0, Status : FAILED java.io.IOException: ORA-00933: SQL command not properly ended at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:289) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-3837: - Attachment: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job. Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221253#comment-13221253 ] Mayank Bansal commented on MAPREDUCE-3837: -- Hi Alejandro Thanks for your help testing this patch, I am really sorry about confusion as I missed one function in the patch. I have attached the new patch , tested it and it is working fine in my local environment. I am not sure how I missed that before. Please let me know if you find any more issues with that. Arun, I believe the issues were in terms of recovering the jobs from the point they crashed. Here what I am doing is very simplistic approach. I am reading the job token file and resubmitting the jobs in case of crash and recover. I am not trying to recover from the point it left from the last run. In this scenario it is a new run of the job and works well. The downside is the whole job will re run however the upside is Users don't need to resubmit the jobs. Please let me know your thoughts. Thanks, Mayank Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job. Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221255#comment-13221255 ] Hudson commented on MAPREDUCE-3896: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1829 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1829/]) MAPREDUCE-3896. Add user information to the delegation token issued by the history server. (Contributed by Vinod Kumar Vavilapalli) (Revision 1296401) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296401 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/krb5.conf pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3896) pig job through oozie hangs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221258#comment-13221258 ] Hudson commented on MAPREDUCE-3896: --- Integrated in Hadoop-Mapreduce-0.23-Commit #629 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/629/]) merge MAPREDUCE-3896 from trunk (Revision 1296402) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/krb5.conf pig job through oozie hangs Key: MAPREDUCE-3896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3896 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-3896-20120228.txt, MAPREDUCE-3896-20120301.txt running pig job on oozie hangs due to race condition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221259#comment-13221259 ] Thomas Graves commented on MAPREDUCE-3954: -- Mostly looks good. And thanks for updating the docs. A couple minor documentation nits. - can you update the comment Environment Variables section at the top of bin/yarn for the new variables - you might add a header to the table in the ClusterSetup - with Daemon | Environment Variable - You didn't change this so its optional. It might be nice in ClusterSetup to say that you configure it size in MB but you leave off the m. Because when I read that I thought I would set the env var to 1000MB. Clean up passing HEAPSIZE to yarn and mapred commands. -- Key: MAPREDUCE-3954 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker Attachments: MR-3954.txt, MR-3954.txt Currently the heap size for all of these is set in yarn-env.sh. JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set it will override JAVA_HEAP_MAX. However, we do not always want to have the RM, NM, and HistoryServer with the exact same heap size. It would be logical to have inside of yarn and mapred to set JAVA_HEAP_MAX if YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug because it is easy to configure the history server to store more entires then the heap can hold. It is also a performance issue if we do not allow the history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3960) web proxy doesn't forward request to AM with configured hostname/IP
[ https://issues.apache.org/jira/browse/MAPREDUCE-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221264#comment-13221264 ] Hadoop QA commented on MAPREDUCE-3960: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516875/MAPREDUCE-3960.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1987//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1987//console This message is automatically generated. web proxy doesn't forward request to AM with configured hostname/IP --- Key: MAPREDUCE-3960 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3960 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: MAPREDUCE-3960.patch If the host the web proxy is running on has an ip alias or similar and the config file is pointing to the hostname that is the aliased ip of the box, the web proxy will send the request from the base ip rather then the aliased ip and the AM will redirect that request to the proxy again instead of accepting it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221271#comment-13221271 ] Alejandro Abdelnur commented on MAPREDUCE-3837: --- I've tested the last patch and works as expected. I'd agree with Mayank that this approach (rerun the full job) seems much less risky than the previous approach (rerun from where it was left). Thus I'm good with the patch as it is much better than what currently is in. Arun, would you reconsider based on the explanation of what Mayank's patch does? Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job. Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3907) Create a mapred-default.xml for the jobhistory server.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated MAPREDUCE-3907:
-
Release Note: Adds documentation for the mentioned properties.
Status: Patch Available (was: Open)
Create a mapred-default.xml for the jobhistory server.
--
Key: MAPREDUCE-3907
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3907
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, security
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Priority: Minor
Attachments: MAPREDUCE-3907.patch
The following configuration properties are documented in
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html#Running_Hadoop_in_Secure_Mode
* mapreduce.jobhistory.address
* mapreduce.jobhistory.keytab
* mapreduce.jobhistory.principal
Create a
{{hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/resources/mapred-default.xml}}
that documents these properties and provides default values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3897) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221303#comment-13221303 ] Thomas Graves commented on MAPREDUCE-3897: -- I should add, my opinion is the other issue with user minimum limit % can be fixed in a separate jira. capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong -- Key: MAPREDUCE-3897 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3897 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3897-1.txt, MAPREDUCE-3897-1.txt, MAPREDUCE-3897-2.txt The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser based on the config yarn.scheduler.capacity.maximum-applications or default 1. MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity), 1) MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100) * userLimitFactor), 1) maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so if max capacity capacity and if you have user limit factor 1 (which is the default) and only 1 user is running, that user will not be allowed to use over the queue capacity, so having it relative to MAX capacity doesn't make sense. That user could easily end up in a deadlock and all its space used by application masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3897) capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221302#comment-13221302 ] Thomas Graves commented on MAPREDUCE-3897: -- +1 for the formula change to fix the current issue described here. I haven't had a chance to review the code in detail. I think there is still a bug in the computation in that it uses the user minimum limit % in the calculation of maxActiveApplicationPerUser, when really that need to factor in the number active users. so similar to computeUserLimit function where it takes the max of userlimit*capacity or capacity/activeusers. Perhaps Arun can comment. capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong -- Key: MAPREDUCE-3897 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3897 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3897-1.txt, MAPREDUCE-3897-1.txt, MAPREDUCE-3897-2.txt The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser based on the config yarn.scheduler.capacity.maximum-applications or default 1. MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity), 1) MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100) * userLimitFactor), 1) maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so if max capacity capacity and if you have user limit factor 1 (which is the default) and only 1 user is running, that user will not be allowed to use over the queue capacity, so having it relative to MAX capacity doesn't make sense. That user could easily end up in a deadlock and all its space used by application masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
Map/ReduceSlotMillis computation incorrect -- Key: MAPREDUCE-3961 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Siddharth Seth Map/ReduceSlot millis are currently computed based on a fixed container size. They should instead be based on the minimum container size offered by the cluster. There's another jira to rename these Counters - based on the resource type. This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3893: -- Attachment: MAPREDUCE-3893-1.txt allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis --- Key: MAPREDUCE-3893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3893-1.txt The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters. I think we should make those configurable on a per queue basis. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3893: -- Status: Patch Available (was: Open) allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis --- Key: MAPREDUCE-3893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3893-1.txt The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters. I think we should make those configurable on a per queue basis. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221344#comment-13221344 ] Konstantin Shvachko commented on MAPREDUCE-3837: I've been reviewing this patch, and have a couple of cosmetic comments below. I agree with Alejandro. This is not introducing new feature, it is just enabling already existing feature. There is low risk, since the feature is enabled in a restricted context, that is restarting failed jobs from scratch rather than trying to continue from the point they were terminated. The patch seems to be larger than it actually is, because it is removing the [troubled] logic responsible for resurrecting the job from its history. Besides that it is simple. Take a look, Arun. Cosmetic comments - Several lines are too long - See several tabs - should be spaces - indentation is wrong in couple of places recoveryManager.addJobForRecovery(JobID.forName(fileName)); shouldRecover = true; // enable actual recovery if num-files 1 - Add spaces after commas in method calls and parameters Otherwise it looks good. Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job. Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3907) Create a mapred-default.xml for the jobhistory server.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221346#comment-13221346
]
Hadoop QA commented on MAPREDUCE-3907:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12515827/MAPREDUCE-3907.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1989//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1989//console
This message is automatically generated.
Create a mapred-default.xml for the jobhistory server.
--
Key: MAPREDUCE-3907
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3907
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, security
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Priority: Minor
Attachments: MAPREDUCE-3907.patch
The following configuration properties are documented in
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html#Running_Hadoop_in_Secure_Mode
* mapreduce.jobhistory.address
* mapreduce.jobhistory.keytab
* mapreduce.jobhistory.principal
Create a
{{hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/resources/mapred-default.xml}}
that documents these properties and provides default values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3962) InverseMapper causes IntWritable type to collected as Text
InverseMapper causes IntWritable type to collected as Text
--
Key: MAPREDUCE-3962
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3962
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Saad Patel
When using the org.apache.hadoop.mapred.lib.InverseMapper, key,values are
inversed as expected. For example (Text,IntWritable) will get inversed.
However, output key,value only works if you use (Text, Text). Below is an
example, where I was chaining 2 jobs:
Job 1:
public static class Reduce extends MapReduceBase implements ReducerText,
IntWritable, Text, IntWritable{...}
Job 2:
conf.setMapperClass(InverseMapper.class);
public static class Reduce extends MapReduceBase implements
ReducerIntWritable, Text, Text, IntWritable {...} //I would expect this to
work. When I do this, I get
java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to
org.apache.hadoop.io.IntWritable
To re-inverse my key,values, I had to do this:
public static class Reduce extends MapReduceBase implements ReducerText, Text,
Text, Text
Notice that in order for the reducer to properly accept key,values, I had to
indicate the key as Text.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221360#comment-13221360 ] Hadoop QA commented on MAPREDUCE-3893: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516901/MAPREDUCE-3893-1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1990//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1990//console This message is automatically generated. allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis --- Key: MAPREDUCE-3893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Eric Payne Priority: Critical Attachments: MAPREDUCE-3893-1.txt The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters. I think we should make those configurable on a per queue basis. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3963) NodeManagers die on startup if they can't connect to the RM
NodeManagers die on startup if they can't connect to the RM --- Key: MAPREDUCE-3963 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3963 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.1 Reporter: Ravi Prakash Priority: Critical Steps to reproduce. Start the NM when the RM is down. The NM tries 10 times, then exits. It should keep trying forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3963) NodeManagers die on startup if they can't connect to the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221368#comment-13221368 ] Ravi Prakash commented on MAPREDUCE-3963: - Thanks to Bach Bui for finding this issue NodeManagers die on startup if they can't connect to the RM --- Key: MAPREDUCE-3963 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3963 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.1 Reporter: Ravi Prakash Priority: Critical Steps to reproduce. Start the NM when the RM is down. The NM tries 10 times, then exits. It should keep trying forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221369#comment-13221369 ] Zhenxiao Luo commented on MAPREDUCE-3952: - @Bhallamudi Yes. Seems the input file is an empty file from execution log: 2012-02-28 15:56:37,219 INFO exec.ExecDriver (ExecDriver.java:addInputPath(829)) - Changed input file to file:/tmp/cloudera/hive_2012-02-28_15-56-37_188_1216173472421796708/-mr-1/1 2012-02-28 15:56:37,226 INFO util.NativeCodeLoader (NativeCodeLoader.java:clinit(50)) - Loaded the native-hadoop library 2012-02-28 15:56:37,610 INFO jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId= 2012-02-28 15:56:37,626 INFO exec.ExecDriver (ExecDriver.java:createTmpDirs(234)) - Making Temp Directory: file:/tmp/cloudera/hive_2012-02-28_15-56-26_431_554636048819260524/-mr-10003 2012-02-28 15:56:37,657 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2012-02-28 15:56:37,684 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(139)) - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2012-02-28 15:56:37,960 WARN snappy.LoadSnappy (LoadSnappy.java:clinit(36)) - Snappy native library is available 2012-02-28 15:56:37,961 INFO snappy.LoadSnappy (LoadSnappy.java:clinit(44)) - Snappy native library loaded 2012-02-28 15:56:37,969 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(370)) - CombineHiveInputSplit creating pool for file:/tmp/cloudera/hive_2012-02-28_15-56-37_188_1216173472421796708/-mr-1/1; using filter path file:/tmp/cloudera/hive_2012-02-28_15-56-37_188_1216173472421796708/-mr-1/1 2012-02-28 15:56:37,970 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 2012-02-28 15:56:37,970 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 2012-02-28 15:56:37,971 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 2012-02-28 15:56:37,971 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 2012-02-28 15:56:37,977 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 1 2012-02-28 15:56:37,982 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(388)) - Arrays.asList iss 2012-02-28 15:56:37,982 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(410)) - iss size: 0 2012-02-28 15:56:37,983 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(417)) - number of splits 0 And, in MR1, the log looks like: 2012-02-28 14:09:54,554 INFO exec.ExecDriver (ExecDriver.java:addInputPath(829)) - Changed input file to file:/tmp/cloudera/hive_2012-02-28_14-09-54_515_1377575814725676804/-mr-1/1 2012-02-28 14:09:54,855 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Initializing JVM Metrics with processName=JobTracker, sessionId= 2012-02-28 14:09:54,871 INFO exec.ExecDriver (ExecDriver.java:createTmpDirs(234)) - Making Temp Directory: file:/tmp/cloudera/hive_2012-02-28_14-09-44_700_3241431154033268523/-mr-10003 2012-02-28 14:09:54,881 WARN mapred.JobClient (JobClient.java:configureCommandLineOptions(539)) - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2012-02-28 14:09:55,037 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(370)) - CombineHiveInputSplit creating pool for file:/tmp/cloudera/hive_2012-02-28_14-09-54_515_1377575814725676804/-mr-1/1; using filter path file:/tmp/cloudera/hive_2012-02-28_14-09-54_515_1377575814725676804/-mr-1/1 2012-02-28 14:09:55,042 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(192)) - Total input paths to process : 1 2012-02-28 14:09:55,056 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(406)) - iss size: 1 2012-02-28 14:09:55,057 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(409)) - adding inputSplitShim into result: Paths:/tmp/cloudera/hive_2012-02-28_14-09-54_515_1377575814725676804/-mr-1/1/emptyFile:0+0 Locations:/default-rack:; InputFormatClass: org.apache.hadoop.mapred.TextInputFormat 2012-02-28 14:09:55,057 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(413)) - number of splits 1 So, in MR1, submitting a
[jira] [Updated] (MAPREDUCE-3935) Annotate Counters.Counter and Counters.Group as @Public
[ https://issues.apache.org/jira/browse/MAPREDUCE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3935: - Resolution: Fixed Fix Version/s: 0.23.3 Target Version/s: 0.23.3 (was: 0.23.3, 1.1.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this. Thanks for taking a look, Robert. Annotate Counters.Counter and Counters.Group as @Public --- Key: MAPREDUCE-3935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3935 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.3 Attachments: MAPREDUCE-3935.patch For clarity these inner classes should be marked as public stable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3935) Annotate Counters.Counter and Counters.Group as @Public
[ https://issues.apache.org/jira/browse/MAPREDUCE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221374#comment-13221374 ] Hudson commented on MAPREDUCE-3935: --- Integrated in Hadoop-Common-trunk-Commit #1823 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1823/]) MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public. (Revision 1296523) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java Annotate Counters.Counter and Counters.Group as @Public --- Key: MAPREDUCE-3935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3935 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.3 Attachments: MAPREDUCE-3935.patch For clarity these inner classes should be marked as public stable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3935) Annotate Counters.Counter and Counters.Group as @Public
[ https://issues.apache.org/jira/browse/MAPREDUCE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221378#comment-13221378 ] Hudson commented on MAPREDUCE-3935: --- Integrated in Hadoop-Hdfs-trunk-Commit #1897 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1897/]) MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public. (Revision 1296523) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java Annotate Counters.Counter and Counters.Group as @Public --- Key: MAPREDUCE-3935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3935 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.3 Attachments: MAPREDUCE-3935.patch For clarity these inner classes should be marked as public stable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221391#comment-13221391 ] Zhenxiao Luo commented on MAPREDUCE-3952: - @Ahmed I think so. Is it the case that, in MR1, an empty file get a split size of 1, and then the record reader will emit no records. And in MR2, an empty file get a split size of 0? In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. --- Key: MAPREDUCE-3952 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Zhenxiao Luo Hive get unexpected result when using MR2(When using MR1, always get expected result). In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. The calling code in Hive, in Hadoop23Shims.java: InputSplit[] splits = super.getSplits(job, numSplits); this get splits.length == 0. In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, numSplits); this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3935) Annotate Counters.Counter and Counters.Group as @Public
[ https://issues.apache.org/jira/browse/MAPREDUCE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221399#comment-13221399 ] Hudson commented on MAPREDUCE-3935: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1830 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1830/]) MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public. (Revision 1296523) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296523 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java Annotate Counters.Counter and Counters.Group as @Public --- Key: MAPREDUCE-3935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3935 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.3 Attachments: MAPREDUCE-3935.patch For clarity these inner classes should be marked as public stable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3935) Annotate Counters.Counter and Counters.Group as @Public
[ https://issues.apache.org/jira/browse/MAPREDUCE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221405#comment-13221405 ] Hudson commented on MAPREDUCE-3935: --- Integrated in Hadoop-Mapreduce-0.23-Commit #630 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/630/]) Merge -r 1296522:1296523 from trunk to branch-23. Fixes: MAPREDUCE-3935. (Revision 1296524) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1296524 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java Annotate Counters.Counter and Counters.Group as @Public --- Key: MAPREDUCE-3935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3935 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Fix For: 0.23.3 Attachments: MAPREDUCE-3935.patch For clarity these inner classes should be marked as public stable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
ResourceManager does not have JVM metrics - Key: MAPREDUCE-3964 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.24.0, 0.23.2 Reporter: Jason Lowe Priority: Critical ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3963) NodeManagers die on startup if they can't connect to the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221494#comment-13221494 ] Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3963: --- Ravi, is this dup of [MAPREDUCE-3676|https://issues.apache.org/jira//browse/MAPREDUCE-3676]? NodeManagers die on startup if they can't connect to the RM --- Key: MAPREDUCE-3963 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3963 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.1 Reporter: Ravi Prakash Priority: Critical Steps to reproduce. Start the NM when the RM is down. The NM tries 10 times, then exits. It should keep trying forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3929) output of mapred -showacl is not clear
[ https://issues.apache.org/jira/browse/MAPREDUCE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3929: --- Status: Patch Available (was: Open) output of mapred -showacl is not clear -- Key: MAPREDUCE-3929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.2, 0.23.3 Reporter: John George Assignee: John George Attachments: MR-3929.patch, MR-3929.patch output of 'mapred queue -showacls' is not very clear. This JIRA is aimed at either fixing that or adding something to the document to make it clear. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3929) output of mapred -showacl is not clear
[ https://issues.apache.org/jira/browse/MAPREDUCE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3929: --- Attachment: MR-3929.patch output of mapred -showacl is not clear -- Key: MAPREDUCE-3929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.2, 0.23.3 Reporter: John George Assignee: John George Attachments: MR-3929.patch, MR-3929.patch output of 'mapred queue -showacls' is not very clear. This JIRA is aimed at either fixing that or adding something to the document to make it clear. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3929) output of mapred -showacl is not clear
[ https://issues.apache.org/jira/browse/MAPREDUCE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221528#comment-13221528 ] John George commented on MAPREDUCE-3929: Thanks for your review Arun. Herewith is attached another patch with unit tests. output of mapred -showacl is not clear -- Key: MAPREDUCE-3929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.2, 0.23.3 Reporter: John George Assignee: John George Attachments: MR-3929.patch, MR-3929.patch output of 'mapred queue -showacls' is not very clear. This JIRA is aimed at either fixing that or adding something to the document to make it clear. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3929) output of mapred -showacl is not clear
[ https://issues.apache.org/jira/browse/MAPREDUCE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221530#comment-13221530 ] Hadoop QA commented on MAPREDUCE-3929: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516928/MR-3929.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1991//console This message is automatically generated. output of mapred -showacl is not clear -- Key: MAPREDUCE-3929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.2, 0.23.3 Reporter: John George Assignee: John George Attachments: MR-3929.patch, MR-3929.patch output of 'mapred queue -showacls' is not very clear. This JIRA is aimed at either fixing that or adding something to the document to make it clear. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
