[jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465434#comment-13465434 ] Tsuyoshi OZAWA commented on MAPREDUCE-4502: --- I've read the overview of the Sailfish paper. 1. It makes IO faster to translate the random access of HDDs to sequential access by batching IO at dumping intermediate files with the original file format(i.e. L-File). This is effective only case that very large intermediate file is generated. 2. Auto-tuning the number of reducers is a good feature for speedup and reducing tuning time, but users cannot control their output. 1. is one optimization of MR jobs with very large scale intermediate data, which is larger than multi tera bytes. It needs to change HDFS to coordinate concurrent access against intermediate files. I wonder if it is commodity case and can pay the trade-offs between the implementation cost and the effect. Do you have any opinion about it? IMHO, 2. is another feature, so we should discuss in another JIRA. Multi-level aggregation with combining the result of maps per node/rack --- Key: MAPREDUCE-4502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: design_v2.pdf, speculative_draft.pdf The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner. This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465436#comment-13465436 ] Tsuyoshi OZAWA commented on MAPREDUCE-4502: --- BTW, I'm starting to implement the aggregation feature with V2 design. Please let me know if you have any opinion about the design note. Thanks. Multi-level aggregation with combining the result of maps per node/rack --- Key: MAPREDUCE-4502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: design_v2.pdf, speculative_draft.pdf The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner. This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4651) Benchmarking random reads with DFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465467#comment-13465467 ] Konstantin Shvachko commented on MAPREDUCE-4651: That is because I committed it. Jakob reviewed. Benchmarking random reads with DFSIO Key: MAPREDUCE-4651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4651 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, test Affects Versions: 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.23.4 Attachments: randomDFSIO.patch, randomDFSIO.patch, randomDFSIO.patch TestDFSIO measures throughput of HDFS write, read, and append operations. It will be useful to have an option to use it for benchmarking random reads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465487#comment-13465487 ] Tom White commented on MAPREDUCE-4278: -- This looks like the right approach. Can you write a unit test that shows that this fixes the problem? You could use a waiting job from UtilsForTests that ensures two jobs are running concurrently. cannot run two local jobs in parallel from the same gateway. Key: MAPREDUCE-4278 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Araceli Henley Attachments: MAPREDUCE-4278-branch1.patch I cannot run two local mode jobs from Pig in parallel from the same gateway, this is a typical use case. If I re-run the tests sequentially, then the test pass. This seems to be a problem from Hadoop. Additionally, the pig harness, expects to be able to run Pig-version-undertest against Pig-version-stable from the same gateway. To replicate the error: I have two clusters running from the same gateway. If I run the Pig regression suites nightly.conf in local mode in paralell - once on each cluster. Conflicts in M/R local mode result in failures in the tests. ERROR1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) at org.apache.hadoop.mapred.Task.done(Task.java:875) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) --- ERROR2: 2012-05-17 20:25:36,762 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache. hadoop.util.LinuxResourceCalculatorPlugin@ffa490e 2012-05-17 20:25:36,837 [Thread-3] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java :153) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm at.java:106) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:489) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) 2012-05-17 20:25:41,291 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-4678:
---
Attachment: MAPREDUCE-4678.patch.txt
The changes look good to me as an improvement but can you add some notes on how
this helps solve negative size exceptions, just so we have it documented?
Your patch has a minor issue that usually arises if you don't (a) use a proper
IDE for Java or (b) don't compile your code before submitting, both of which
are good practices to follow :)
Sadly, Java isn't Python, so please wrap () around your if statements and also
use { } in the structure (for code convention), like so:
{code}
+ if ((args[i]).equalsIgnoreCase(-depth)) {
+depth = Integer.parseInt(args[i++].trim());
+ } else if ((args[i]).equalsIgnoreCase(-height)) {
+ height = Integer.parseInt(args[i++].trim());
+ } else if ((args[i]).equalsIgnoreCase(-width)) {
+ width = Integer.parseInt(args[i++].trim());
+ }
{code}
And then please submit a new patch (after making sure it at least compiles
locally! {{mvn compile}}) :)
Thanks much for reporting this in Chris, and for working on the patch.
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Chris McConnell
Priority: Minor
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4651) Benchmarking random reads with DFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465601#comment-13465601 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-4651: --- Oops, I forgot to reload the page. It is great that Jakob has reviewed it. Benchmarking random reads with DFSIO Key: MAPREDUCE-4651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4651 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, test Affects Versions: 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.23.4 Attachments: randomDFSIO.patch, randomDFSIO.patch, randomDFSIO.patch TestDFSIO measures throughput of HDFS write, read, and append operations. It will be useful to have an option to use it for benchmarking random reads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3223: --- Attachment: mr-3223.txt Thanks Robert, rebased the patch. Will commit once I get a new +1 from jenkins. Remove MR1 configs from mapred-default.xml -- Key: MAPREDUCE-3223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt All of the MRv1 configs are still in mapred-default.xml. This is confusing when trying to make config changes. Since a lot of the input/output format tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4692) Investigate and remove MR1 JTConfig and its constants use in the MR project on trunk
Harsh J created MAPREDUCE-4692:
--
Summary: Investigate and remove MR1 JTConfig and its constants use
in the MR project on trunk
Key: MAPREDUCE-4692
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4692
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Harsh J
Priority: Minor
Filed on behalf of Robert from MAPREDUCE-3223
{code}
Are there any JIRAs to deprecate the configs from where they reside in the
code?
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
for example. I know we cannot delete them out just yet, because MRV1 code
still exists and may be using it, but it would be good to mark all of those
configs as deprecated. So that we can delete them in trunk once the MRV1 code
is completely removed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465607#comment-13465607 ] Harsh J commented on MAPREDUCE-3223: Filed MAPREDUCE-4692 to address the mentioned source-side of concern. Remove MR1 configs from mapred-default.xml -- Key: MAPREDUCE-3223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt All of the MRv1 configs are still in mapred-default.xml. This is confusing when trying to make config changes. Since a lot of the input/output format tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465620#comment-13465620
]
Hadoop QA commented on MAPREDUCE-3223:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12546990/mr-3223.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
+0 tests included. The patch appears to be a documentation patch that
doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2888//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2888//console
This message is automatically generated.
Remove MR1 configs from mapred-default.xml
--
Key: MAPREDUCE-3223
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation, mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt
All of the MRv1 configs are still in mapred-default.xml. This is confusing
when trying to make config changes. Since a lot of the input/output format
tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml
for now, and once that dependency is broken, we can remove them entirely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3223: --- Resolution: Fixed Fix Version/s: 3.0.0 Target Version/s: (was: ) Status: Resolved (was: Patch Available) I committed this to trunk. Thanks for the base patch Todd, and thanks for reviewing Robert! Remove MR1 configs from mapred-default.xml -- Key: MAPREDUCE-3223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0 Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt All of the MRv1 configs are still in mapred-default.xml. This is confusing when trying to make config changes. Since a lot of the input/output format tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465664#comment-13465664 ] Hudson commented on MAPREDUCE-3223: --- Integrated in Hadoop-Hdfs-trunk-Commit #2845 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2845/]) MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml. Contributed by Todd Lipcon (harsh) (Revision 1391498) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391498 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml Remove MR1 configs from mapred-default.xml -- Key: MAPREDUCE-3223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0 Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt All of the MRv1 configs are still in mapred-default.xml. This is confusing when trying to make config changes. Since a lot of the input/output format tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465665#comment-13465665 ] Hudson commented on MAPREDUCE-3223: --- Integrated in Hadoop-Common-trunk-Commit #2782 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2782/]) MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml. Contributed by Todd Lipcon (harsh) (Revision 1391498) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391498 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml Remove MR1 configs from mapred-default.xml -- Key: MAPREDUCE-3223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0 Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt All of the MRv1 configs are still in mapred-default.xml. This is confusing when trying to make config changes. Since a lot of the input/output format tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465666#comment-13465666 ] Hudson commented on MAPREDUCE-3223: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2804 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2804/]) MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml. Contributed by Todd Lipcon (harsh) (Revision 1391498) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391498 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml Remove MR1 configs from mapred-default.xml -- Key: MAPREDUCE-3223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0 Attachments: mr-3223.txt, mr-3223.txt, mr-3223.txt All of the MRv1 configs are still in mapred-default.xml. This is confusing when trying to make config changes. Since a lot of the input/output format tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: MAPREDUCE-4678.patch Patch version 2 -- sorry about that Harsh! Cheers! Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reassigned MAPREDUCE-4691:
--
Assignee: Robert Joseph Evans
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: (was: MAPREDUCE-4678.patch) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: (was: MAPREDUCE-4678.patch) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: MAPREDUCE-4678.patch Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4692) Investigate and remove MR1 JTConfig and its constants use in the MR project on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated MAPREDUCE-4692:
---
Description:
Filed on behalf of Robert from MAPREDUCE-3223
{quote}
Are there any JIRAs to deprecate the configs from where they reside in the
code?
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
for example. I know we cannot delete them out just yet, because MRV1 code
still exists and may be using it, but it would be good to mark all of those
configs as deprecated. So that we can delete them in trunk once the MRV1 code
is completely removed.
{quote}
was:
Filed on behalf of Robert from MAPREDUCE-3223
{code}
Are there any JIRAs to deprecate the configs from where they reside in the
code?
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
for example. I know we cannot delete them out just yet, because MRV1 code
still exists and may be using it, but it would be good to mark all of those
configs as deprecated. So that we can delete them in trunk once the MRV1 code
is completely removed.
{code}
Investigate and remove MR1 JTConfig and its constants use in the MR project
on trunk
Key: MAPREDUCE-4692
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4692
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Harsh J
Priority: Minor
Filed on behalf of Robert from MAPREDUCE-3223
{quote}
Are there any JIRAs to deprecate the configs from where they reside in the
code?
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
for example. I know we cannot delete them out just yet, because MRV1 code
still exists and may be using it, but it would be good to mark all of those
configs as deprecated. So that we can delete them in trunk once the MRV1 code
is completely removed.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4691:
---
Attachment: MR-4691.txt
This patch should fix the issue. Recreating the problem is not that simple in
the form of a unit tests, so I have not added any now ones in. I am going to
manually verify the issue, but I wanted to kick jenkins in parallel.
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4678: --- Component/s: examples Target Version/s: 3.0.0 Affects Version/s: (was: 0.20.205.0) (was: 0.23.0) 2.0.0-alpha Hadoop Flags: Reviewed Thanks for the fixed patch. I built a jar and I can confirm the default behavior is no longer running into a negative size exception. Thanks for also improving the CLI opts style. +1, committing shortly. Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 2.0.0-alpha Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-4554: Status: Open (was: Patch Available) Job Credentials are not transmitted if security is turned off - Key: MAPREDUCE-4554 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Affects Versions: 2.0.0-alpha Reporter: Benoy Antony Assignee: Benoy Antony Fix For: trunk Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch Credentials (secret keys) can be passed to a job via mapreduce.job.credentials.json or mapreduce.job.credentials.binary . These credentials get submitted during job submission and are made available to the task processes. In HADOOP 1, these credentials get submitted and routed to task processes even if security was off. In HADOOP 2 , these credentials are transmitted only when the security is turned on. This should be changed for two reasons: 1) It is not backward compatible. 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-4554: Attachment: MR_4554_trunk.patch fixing the spacing issue on the new files Job Credentials are not transmitted if security is turned off - Key: MAPREDUCE-4554 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Affects Versions: 2.0.0-alpha Reporter: Benoy Antony Assignee: Benoy Antony Fix For: trunk Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch Credentials (secret keys) can be passed to a job via mapreduce.job.credentials.json or mapreduce.job.credentials.binary . These credentials get submitted during job submission and are made available to the task processes. In HADOOP 1, these credentials get submitted and routed to task processes even if security was off. In HADOOP 2 , these credentials are transmitted only when the security is turned on. This should be changed for two reasons: 1) It is not backward compatible. 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-4678. Resolution: Fixed Fix Version/s: 3.0.0 Target Version/s: (was: 3.0.0) Thanks for the patch Chris, I've committed it to trunk. Hope to see more contributions in the future :) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 2.0.0-alpha Reporter: Chris McConnell Priority: Minor Fix For: 3.0.0 Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-4678:
---
Description:
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
was:
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 2.0.0-alpha
Reporter: Chris McConnell
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-4554: Status: Patch Available (was: Open) Job Credentials are not transmitted if security is turned off - Key: MAPREDUCE-4554 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Affects Versions: 2.0.0-alpha Reporter: Benoy Antony Assignee: Benoy Antony Fix For: trunk Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch Credentials (secret keys) can be passed to a job via mapreduce.job.credentials.json or mapreduce.job.credentials.binary . These credentials get submitted during job submission and are made available to the task processes. In HADOOP 1, these credentials get submitted and routed to task processes even if security was off. In HADOOP 2 , these credentials are transmitted only when the security is turned on. This should be changed for two reasons: 1) It is not backward compatible. 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465731#comment-13465731
]
Hudson commented on MAPREDUCE-4678:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #2805 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2805/])
MAPREDUCE-4678. Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException. Contributed by Chris McConnell. (harsh)
(Revision 1391551)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391551
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/dancing/DistributedPentomino.java
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 2.0.0-alpha
Reporter: Chris McConnell
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465730#comment-13465730
]
Hudson commented on MAPREDUCE-4678:
---
Integrated in Hadoop-Hdfs-trunk-Commit #2846 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2846/])
MAPREDUCE-4678. Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException. Contributed by Chris McConnell. (harsh)
(Revision 1391551)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391551
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/dancing/DistributedPentomino.java
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 2.0.0-alpha
Reporter: Chris McConnell
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers reassigned MAPREDUCE-4678:
-
Assignee: Chris McConnell
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 2.0.0-alpha
Reporter: Chris McConnell
Assignee: Chris McConnell
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465733#comment-13465733
]
Hudson commented on MAPREDUCE-4678:
---
Integrated in Hadoop-Common-trunk-Commit #2783 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2783/])
MAPREDUCE-4678. Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException. Contributed by Chris McConnell. (harsh)
(Revision 1391551)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391551
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/dancing/DistributedPentomino.java
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 2.0.0-alpha
Reporter: Chris McConnell
Assignee: Chris McConnell
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465738#comment-13465738 ] Sandy Ryza commented on MAPREDUCE-4278: --- If the jobs are in the same process they're already prevented from colliding by the (not sure what to call it) 0001 part of the job id. Do you have any advice on how to test it in light of this? Also, I realized that with my approach the randids could get mixed if two jobs were submitted concurrently using the same LocalJobRunner. Is this a concern? cannot run two local jobs in parallel from the same gateway. Key: MAPREDUCE-4278 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Araceli Henley Attachments: MAPREDUCE-4278-branch1.patch I cannot run two local mode jobs from Pig in parallel from the same gateway, this is a typical use case. If I re-run the tests sequentially, then the test pass. This seems to be a problem from Hadoop. Additionally, the pig harness, expects to be able to run Pig-version-undertest against Pig-version-stable from the same gateway. To replicate the error: I have two clusters running from the same gateway. If I run the Pig regression suites nightly.conf in local mode in paralell - once on each cluster. Conflicts in M/R local mode result in failures in the tests. ERROR1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) at org.apache.hadoop.mapred.Task.done(Task.java:875) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) --- ERROR2: 2012-05-17 20:25:36,762 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache. hadoop.util.LinuxResourceCalculatorPlugin@ffa490e 2012-05-17 20:25:36,837 [Thread-3] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java :153) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm at.java:106) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:489) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) 2012-05-17 20:25:41,291 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[
https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465789#comment-13465789
]
Hadoop QA commented on MAPREDUCE-4554:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547009/MR_4554_trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.lib.aggregate.TestAggregates
org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat
org.apache.hadoop.mapred.TestTaskCommit
org.apache.hadoop.mapred.TestFieldSelection
org.apache.hadoop.mapreduce.TestLocalRunner
org.apache.hadoop.mapred.TestUserDefinedCounters
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.TestMROutputFormat
org.apache.hadoop.mapred.TestLineRecordReader
org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection
org.apache.hadoop.mapred.TestLocalMRNotification
org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
org.apache.hadoop.mapred.lib.TestChainMapReduce
org.apache.hadoop.mapreduce.TestMapReduce
org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge
org.apache.hadoop.mapred.lib.TestKeyFieldBasedComparator
org.apache.hadoop.mapred.lib.TestMultithreadedMapRunner
org.apache.hadoop.mapred.lib.TestMultipleOutputs
org.apache.hadoop.mapred.TestJavaSerialization
org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader
org.apache.hadoop.mapreduce.lib.output.TestMRMultipleOutputs
org.apache.hadoop.mapred.TestCollect
org.apache.hadoop.fs.slive.TestSlive
org.apache.hadoop.mapred.join.TestDatamerge
org.apache.hadoop.mapreduce.TestMapCollection
org.apache.hadoop.fs.TestDFSIO
org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates
org.apache.hadoop.mapred.TestMapRed
org.apache.hadoop.mapred.TestFileOutputFormat
org.apache.hadoop.mapreduce.TestValueIterReset
org.apache.hadoop.mapred.TestMapOutputType
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.fs.TestFileSystem
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapred.TestReporter
org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator
org.apache.hadoop.mapreduce.lib.chain.TestChainErrors
org.apache.hadoop.mapreduce.lib.chain.TestSingleElementChain
org.apache.hadoop.mapreduce.lib.input.TestMultipleInputs
org.apache.hadoop.mapred.TestComparators
org.apache.hadoop.mapreduce.lib.chain.TestMapReduceChain
org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2889//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2889//console
This message is automatically generated.
Job Credentials are not transmitted if security is turned off
-
Key: MAPREDUCE-4554
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, security
Affects Versions: 2.0.0-alpha
Reporter: Benoy Antony
[jira] [Updated] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4691:
---
Target Version/s: 0.23.4
Status: Patch Available (was: Open)
Oops forgot to submit the patch :)
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 2.0.1-alpha, 0.23.3
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465835#comment-13465835
]
Hadoop QA commented on MAPREDUCE-4691:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547008/MR-4691.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2890//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2890//console
This message is automatically generated.
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465846#comment-13465846
]
Robert Joseph Evans commented on MAPREDUCE-4691:
From my manual tests it looks like this patch does indeed solve the issue at
hand.
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4689:
--
Attachment: MAPREDUCE-4689.patch
Patch to CompletedTask so it returns empty counters instead of null counters in
the task report.
In addition to unit test, I also manually tested it with a fail job and
verified that JobClient.getMapTaskReports did not throw an NPE when it
redirected to the historyserver for the failed job.
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4689:
--
Assignee: Jason Lowe
Target Version/s: 0.23.4, 2.0.3-alpha
Status: Patch Available (was: Open)
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465922#comment-13465922
]
Jason Lowe commented on MAPREDUCE-4691:
---
+1 lgtm, will commit this shortly.
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465923#comment-13465923
]
Hadoop QA commented on MAPREDUCE-4689:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547047/MAPREDUCE-4689.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2891//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2891//console
This message is automatically generated.
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4676) Add test for job history cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4676: -- Affects Version/s: (was: 0.20.2) 1.0.3 2.0.1-alpha Add test for job history cleaner Key: MAPREDUCE-4676 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3, 2.0.1-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4676.patch Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up the job history -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465935#comment-13465935
]
Robert Joseph Evans commented on MAPREDUCE-4689:
The patch looks OK to me +1. But this seems more like a band-aide than a full
solution. Shouldn't we be returning counters for the failed tasks too, just
like the AM does?
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4676) Add test for job history cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4676: -- Attachment: MAPREDUCE-4676-trunk.patch Add test for job history cleaner Key: MAPREDUCE-4676 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3, 2.0.1-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4676.patch, MAPREDUCE-4676-trunk.patch Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up the job history -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4676) Add test for job history cleaner
[
https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465938#comment-13465938
]
Hadoop QA commented on MAPREDUCE-4676:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547056/MAPREDUCE-4676-trunk.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2892//console
This message is automatically generated.
Add test for job history cleaner
Key: MAPREDUCE-4676
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 1.0.3, 2.0.1-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4676.patch, MAPREDUCE-4676-trunk.patch
Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up
the job history
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465941#comment-13465941
]
Hudson commented on MAPREDUCE-4691:
---
Integrated in Hadoop-Hdfs-trunk-Commit #2848 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2848/])
MAPREDUCE-4691. Historyserver can report Unknown job after RM says job
has completed. Contributed by Robert Joseph Evans. (Revision 1391671)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391671
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Fix For: 3.0.0, 0.23.4, 2.0.3-alpha
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4691:
--
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
0.23.4
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, Bobby. I merged this into trunk, branch-2, and branch-0.23.
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Fix For: 3.0.0, 0.23.4, 2.0.3-alpha
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465943#comment-13465943
]
Hudson commented on MAPREDUCE-4691:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #2807 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2807/])
MAPREDUCE-4691. Historyserver can report Unknown job after RM says job
has completed. Contributed by Robert Joseph Evans. (Revision 1391671)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391671
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Fix For: 3.0.0, 0.23.4, 2.0.3-alpha
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4691) Historyserver can report Unknown job after RM says job has completed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465944#comment-13465944
]
Hudson commented on MAPREDUCE-4691:
---
Integrated in Hadoop-Common-trunk-Commit #2785 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2785/])
MAPREDUCE-4691. Historyserver can report Unknown job after RM says job
has completed. Contributed by Robert Joseph Evans. (Revision 1391671)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391671
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
Historyserver can report Unknown job after RM says job has completed
--
Key: MAPREDUCE-4691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4691
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Robert Joseph Evans
Priority: Critical
Fix For: 3.0.0, 0.23.4, 2.0.3-alpha
Attachments: MR-4691.txt
Example traceback from the client:
{noformat}
2012-09-27 20:28:38,068 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-09-27 20:28:38,530 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Error from remote end:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,530 [main] ERROR
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:xxx (auth:KERBEROS)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
2012-09-27 20:28:38,531 [main] WARN org.apache.pig.tools.pigstats.JobStats -
Failed to get map task report
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unknown job job_1348097917603_3019
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy11.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:116)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:482)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4693) Historyserver should provide counters for failed tasks
Jason Lowe created MAPREDUCE-4693: - Summary: Historyserver should provide counters for failed tasks Key: MAPREDUCE-4693 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4693 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Reporter: Jason Lowe Currently the historyserver is not providing counters for failed tasks, even though they are available via the AM as long as the job is still running. Those counters are lost when the client needs to redirect to the historyserver after the job completes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4676) Add test for job history cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4676: -- Attachment: (was: MAPREDUCE-4676-trunk.patch) Add test for job history cleaner Key: MAPREDUCE-4676 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3, 2.0.1-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4676.patch Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up the job history -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465956#comment-13465956
]
Hudson commented on MAPREDUCE-4689:
---
Integrated in Hadoop-Hdfs-trunk-Commit #2849 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2849/])
MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE
(jlowe via bobby) (Revision 1391679)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391679
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465958#comment-13465958
]
Hudson commented on MAPREDUCE-4689:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #2808 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2808/])
MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE
(jlowe via bobby) (Revision 1391679)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391679
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465963#comment-13465963
]
Hudson commented on MAPREDUCE-4689:
---
Integrated in Hadoop-Common-trunk-Commit #2786 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2786/])
MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE
(jlowe via bobby) (Revision 1391679)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1391679
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4676) Add test for job history cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4676: -- Attachment: MAPREDUCE-4676-trunk.patch Add test for job history cleaner Key: MAPREDUCE-4676 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3, 2.0.1-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4676.patch, MAPREDUCE-4676-trunk.patch Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up the job history -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4689) JobClient.getMapTaskReports on failed job results in NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4689:
---
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
0.23.4
3.0.0
Status: Resolved (was: Patch Available)
Thanks Jason,
I put this into branch-2, branch-0.23, and trunk
JobClient.getMapTaskReports on failed job results in NPE
Key: MAPREDUCE-4689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Fix For: 3.0.0, 0.23.4, 2.0.3-alpha
Attachments: MAPREDUCE-4689.patch
When calling JobClient.getMapTaskReports for a job that has failed results in
an NPE. For example:
{noformat}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.init(AbstractCounters.java:107)
at org.apache.hadoop.mapred.Counters.init(Counters.java:71)
at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:80)
at org.apache.hadoop.mapred.TaskReport.downgrade(TaskReport.java:81)
at
org.apache.hadoop.mapred.TaskReport.downgradeArray(TaskReport.java:88)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:691)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:681)
...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4676) Add test for job history cleaner
[
https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465983#comment-13465983
]
Hadoop QA commented on MAPREDUCE-4676:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547061/MAPREDUCE-4676-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2893//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2893//console
This message is automatically generated.
Add test for job history cleaner
Key: MAPREDUCE-4676
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 1.0.3, 2.0.1-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4676.patch, MAPREDUCE-4676-trunk.patch
Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up
the job history
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4475) testForEmptyFile failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan resolved MAPREDUCE-4475. - Resolution: Duplicate testForEmptyFile failed --- Key: MAPREDUCE-4475 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4475 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Eli Collins This test failed in a recent test run: testForEmptyFile(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat): expected:0 but was:1 https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1146/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
