[jira] [Commented] (MAPREDUCE-2559) ant binary fails due to missing c++ lib dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045297#comment-13045297 ] Eli Collins commented on MAPREDUCE-2559: I think the findbugs and test failures are unrelated to this patch. Eric, Todd - does this patch and the one on HDFS-2040 look reasonable to you guys? > ant binary fails due to missing c++ lib dir > --- > > Key: MAPREDUCE-2559 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2559 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.20.3 > Environment: Redhat 5.5, Java 6 >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2559-1.patch, MAPREDUCE-2559-2.patch, > MAPREDUCE-2559-3.patch, MAPREDUCE-2559.patch, mapreduce-2559-4.patch > > > Post MAPRED-2521 ant binary fails without "-Dcompile.c++=true > -Dcompile.native=true". The bin-package is trying to copy from the c++ lib > dir which doesn't exist yet. The binary target should check for the existence > of this dir or would also be reasonable to depend on the compile-c++ (since > this is the binary target). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when readFields() called
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045269#comment-13045269 ] Hadoop QA commented on MAPREDUCE-2557: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481646/MAPREDUCE-2557-style.patch against trunk revision 1132807. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/354//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/354//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/354//console This message is automatically generated. > Counters don't reset state when readFields() called > --- > > Key: MAPREDUCE-2557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: William Slacum >Priority: Trivial > Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > When calling readFields() on a Counters object, the internal state is not > completely reset. The IdentityHashMap, Counter> cache retains all > previous mappings, even after the actual CounterGroups are changed. Using the > same Counters pointer over and over again results in the cache always keeping > the mapping for the first call to getCounter(Enum). I've add a clear() > call to the cache when readFields() is called and added a unit test to verify > that it works. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2549) Potential resource leaks in HadoopServer.java, RunOnHadoopWizard.java and Environment.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045265#comment-13045265 ] Devaraj K commented on MAPREDUCE-2549: -- Hi Todd, I prepared eclipse plug-in by applying this patch and tested it manually. > Potential resource leaks in HadoopServer.java, RunOnHadoopWizard.java and > Environment.java > -- > > Key: MAPREDUCE-2549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/eclipse-plugin, contrib/streaming >Affects Versions: 0.23.0 >Reporter: Devaraj K >Assignee: Devaraj K > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2549.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2571) CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException
[ https://issues.apache.org/jira/browse/MAPREDUCE-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bochun Bai updated MAPREDUCE-2571: -- Status: Patch Available (was: Open) The patch uses a loop instead of toArray in getSplits. > CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException > --- > > Key: MAPREDUCE-2571 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2571 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Bochun Bai >Priority: Blocker > Attachments: MAPREDUCE-2571.patch > > > The getSplits methods of > org.apache.hadoop.mapred.lib.CombineFileInputFormat > not work. > ...mapred.lib.CombineFileInputFormat(0.20-style) is a proxy for > ...mapreduce.lib.input.CombineFileInputFormat(0.21-style) > The 0.21-style getSplits returns > ArrayList<...mapreduce.lib.input.CombineFileSplit> > and the 0.20-style delegation calls toArray(...mapred.InputSplit[]) > The ...mapreduce.lib.input.CombineFileSplit is based on > ...mapreduce.InputSplit > and ...mapred.InputSplit is a interface, not a super-class of > ...mapreduce.InputSplit -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2571) CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException
[ https://issues.apache.org/jira/browse/MAPREDUCE-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bochun Bai updated MAPREDUCE-2571: -- Attachment: MAPREDUCE-2571.patch > CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException > --- > > Key: MAPREDUCE-2571 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2571 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Bochun Bai >Priority: Blocker > Attachments: MAPREDUCE-2571.patch > > > The getSplits methods of > org.apache.hadoop.mapred.lib.CombineFileInputFormat > not work. > ...mapred.lib.CombineFileInputFormat(0.20-style) is a proxy for > ...mapreduce.lib.input.CombineFileInputFormat(0.21-style) > The 0.21-style getSplits returns > ArrayList<...mapreduce.lib.input.CombineFileSplit> > and the 0.20-style delegation calls toArray(...mapred.InputSplit[]) > The ...mapreduce.lib.input.CombineFileSplit is based on > ...mapreduce.InputSplit > and ...mapred.InputSplit is a interface, not a super-class of > ...mapreduce.InputSplit -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2571) CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException
CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException --- Key: MAPREDUCE-2571 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2571 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Bochun Bai Priority: Blocker The getSplits methods of org.apache.hadoop.mapred.lib.CombineFileInputFormat not work. ...mapred.lib.CombineFileInputFormat(0.20-style) is a proxy for ...mapreduce.lib.input.CombineFileInputFormat(0.21-style) The 0.21-style getSplits returns ArrayList<...mapreduce.lib.input.CombineFileSplit> and the 0.20-style delegation calls toArray(...mapred.InputSplit[]) The ...mapreduce.lib.input.CombineFileSplit is based on ...mapreduce.InputSplit and ...mapred.InputSplit is a interface, not a super-class of ...mapreduce.InputSplit -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2568) Fix TestFileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045244#comment-13045244 ] Hadoop QA commented on MAPREDUCE-2568: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481604/MAPREDUCE-2568.patch against trunk revision 1132807. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/353//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/353//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/353//console This message is automatically generated. > Fix TestFileSystem > -- > > Key: MAPREDUCE-2568 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2568 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: MAPREDUCE-2568.patch > > > CommandFormat which is used to parse command lines is following posix > conventions of stopping at the first non-argument. There is just one test in > this file which placed an option in the middle of the args. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2559) ant binary fails due to missing c++ lib dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045239#comment-13045239 ] Hadoop QA commented on MAPREDUCE-2559: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481640/mapreduce-2559-4.patch against trunk revision 1132807. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//console This message is automatically generated. > ant binary fails due to missing c++ lib dir > --- > > Key: MAPREDUCE-2559 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2559 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.20.3 > Environment: Redhat 5.5, Java 6 >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2559-1.patch, MAPREDUCE-2559-2.patch, > MAPREDUCE-2559-3.patch, MAPREDUCE-2559.patch, mapreduce-2559-4.patch > > > Post MAPRED-2521 ant binary fails without "-Dcompile.c++=true > -Dcompile.native=true". The bin-package is trying to copy from the c++ lib > dir which doesn't exist yet. The binary target should check for the existence > of this dir or would also be reasonable to depend on the compile-c++ (since > this is the binary target). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2557) Counters don't reset state when readFields() called
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Slacum updated MAPREDUCE-2557: -- Attachment: MAPREDUCE-2557-style.patch Made the "L" next to long's capital for easier viewing. > Counters don't reset state when readFields() called > --- > > Key: MAPREDUCE-2557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: William Slacum >Priority: Trivial > Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > When calling readFields() on a Counters object, the internal state is not > completely reset. The IdentityHashMap, Counter> cache retains all > previous mappings, even after the actual CounterGroups are changed. Using the > same Counters pointer over and over again results in the cache always keeping > the mapping for the first call to getCounter(Enum). I've add a clear() > call to the cache when readFields() is called and added a unit test to verify > that it works. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-143) OOM in the TaskTracker while serving map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045237#comment-13045237 ] Liyin Liang commented on MAPREDUCE-143: --- bq. I think we ran into the same issue, any work around or config tweak to avoid running into this? Thanks. I have created MAPREDUCE-2510 for this problem. As Chris's comment, Jetty 6.1.26 does not have this behavior. However, Jetty 6.1.26 has its own bugs MAPREDUCE-2529 and MAPREDUCE-2530 which are more serious than OOM. > OOM in the TaskTracker while serving map outputs > > > Key: MAPREDUCE-143 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-143 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Devaraj Das > > Saw this exception in the TT logs: > 2009-02-06 06:18:08,553 ERROR org.mortbay.log: EXCEPTION > java.lang.OutOfMemoryError: GC overhead limit exceeded > 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput > java.lang.OutOfMemoryError: GC overhead limit exceeded > 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39) > at java.nio.ByteBuffer.allocate(ByteBuffer.java:312) > at > org.mortbay.io.nio.IndirectNIOBuffer.(IndirectNIOBuffer.java:28) > at > org.mortbay.jetty.nio.AbstractNIOConnector.newBuffer(AbstractNIOConnector.java:71) > at > org.mortbay.jetty.AbstractBuffers.getBuffer(AbstractBuffers.java:131) > at org.mortbay.jetty.HttpGenerator.addContent(HttpGenerator.java:145) > at > org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:642) > at > org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577) > at > org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2879) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when readFields() called
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045236#comment-13045236 ] William Slacum commented on MAPREDUCE-2557: --- Hey Todd, 1) No problem on that 2) I think in terms of running a MR job, there's no real effect unless the framework is re-using the same Counters reference for different jobs, which could happen if JVM re-use is enabled. I encountered this while writing an MR job that used Counters as an input value type. I had written my own InputFormat that re-used a Counters reference and found I was getting odd values when performing operations on Counter values. > Counters don't reset state when readFields() called > --- > > Key: MAPREDUCE-2557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: William Slacum >Priority: Trivial > Attachments: MAPREDUCE-2557.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > When calling readFields() on a Counters object, the internal state is not > completely reset. The IdentityHashMap, Counter> cache retains all > previous mappings, even after the actual CounterGroups are changed. Using the > same Counters pointer over and over again results in the cache always keeping > the mapping for the first call to getCounter(Enum). I've add a clear() > call to the cache when readFields() is called and added a unit test to verify > that it works. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2487) ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
[ https://issues.apache.org/jira/browse/MAPREDUCE-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045231#comment-13045231 ] Hudson commented on MAPREDUCE-2487: --- Integrated in Hadoop-Mapreduce-22-branch #58 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/58/]) MAPREDUCE-2487. Amend previous commit with the new test case (forgot to svn add) MAPREDUCE-2487. ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE. Contributed by Devaraj K. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132721 Files : * /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/mapred/lib/TestChain.java todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132720 Files : * /hadoop/mapreduce/branches/branch-0.22/CHANGES.txt * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapred/lib/Chain.java > ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE > - > > Key: MAPREDUCE-2487 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2487 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Forrest Vines >Assignee: Devaraj K >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2487.patch > > > On line 293 of o.a.h.mapred.lib.Chain in setReducer(...): > reducerConf.setBoolean(MAPPER_BY_VALUE, byValue); > this should be REDUCER_BY_VALUE. > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2559) ant binary fails due to missing c++ lib dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2559: --- Attachment: mapreduce-2559-4.patch I filed HDFS-2040 to make HDFS more consistent with Common and MR (where ant binary only builds native code if passed a flag). See that jira for rationale. Attaching mapreduce-2559-4.patch, minimal patch to allow ant binary to build the tarball as it did previously. > ant binary fails due to missing c++ lib dir > --- > > Key: MAPREDUCE-2559 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2559 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.20.3 > Environment: Redhat 5.5, Java 6 >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2559-1.patch, MAPREDUCE-2559-2.patch, > MAPREDUCE-2559-3.patch, MAPREDUCE-2559.patch, mapreduce-2559-4.patch > > > Post MAPRED-2521 ant binary fails without "-Dcompile.c++=true > -Dcompile.native=true". The bin-package is trying to copy from the c++ lib > dir which doesn't exist yet. The binary target should check for the existence > of this dir or would also be reasonable to depend on the compile-c++ (since > this is the binary target). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045186#comment-13045186
]
Hudson commented on MAPREDUCE-2185:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #717 (See
[https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/717/])
MAPREDUCE-2185. Fix infinite loop at creating splits using
CombineFileInputFormat. (Ramkumar Vadali via schen)
schen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132807
Files :
* /hadoop/mapreduce/trunk/CHANGES.txt
*
/hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
*
/hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
> Infinite loop at creating splits using CombineFileInputFormat
> -
>
> Key: MAPREDUCE-2185
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Hairong Kuang
>Assignee: Ramkumar Vadali
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2185.patch
>
>
> This is caused by a missing block in HDFS. So the block's locations are
> empty. The following code adds the block to blockToNodes map but not to
> rackToBlocks map. Later on when generating splits, only blocks in
> rackToBlocks are removed from blockToNodes map. So blockToNodes map can never
> become empty therefore causing infinite loop
> {code}
> // add this block to the block --> node locations map
> blockToNodes.put(oneblock, oneblock.hosts);
> // add this block to the rack --> block map
> for (int j = 0; j < oneblock.racks.length; j++) {
> ..
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Chen updated MAPREDUCE-2185:
--
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I have committed this. Thanks Ram for the fix and dhruba for the review.
> Infinite loop at creating splits using CombineFileInputFormat
> -
>
> Key: MAPREDUCE-2185
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Hairong Kuang
>Assignee: Ramkumar Vadali
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2185.patch
>
>
> This is caused by a missing block in HDFS. So the block's locations are
> empty. The following code adds the block to blockToNodes map but not to
> rackToBlocks map. Later on when generating splits, only blocks in
> rackToBlocks are removed from blockToNodes map. So blockToNodes map can never
> become empty therefore causing infinite loop
> {code}
> // add this block to the block --> node locations map
> blockToNodes.put(oneblock, oneblock.hosts);
> // add this block to the rack --> block map
> for (int j = 0; j < oneblock.racks.length; j++) {
> ..
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2568) Fix TestFileSystem
[
https://issues.apache.org/jira/browse/MAPREDUCE-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045160#comment-13045160
]
Daryn Sharp commented on MAPREDUCE-2568:
Good question. Yes, there are positive and negative tests in hadoop common. I
question why MR is even testing a class ({{CommandFormat}}) that it doesn't
"own"...
> Fix TestFileSystem
> --
>
> Key: MAPREDUCE-2568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2568
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: MAPREDUCE-2568.patch
>
>
> CommandFormat which is used to parse command lines is following posix
> conventions of stopping at the first non-argument. There is just one test in
> this file which placed an option in the middle of the args.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas resolved MAPREDUCE-2535.
--
Resolution: Fixed
Committed the follow-up. Thanks for the quick fix.
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch, MR-2535-failures-v1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045112#comment-13045112
]
Robert Joseph Evans commented on MAPREDUCE-2535:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 18 new or
modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1
warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to
differ from the contents of the lib directories.
The Eclipse issue and the Javadocs issues are bogus. I didn't even change and
of the source, just tests.
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch, MR-2535-failures-v1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-2535:
---
Attachment: MR-2535-failures-v1.patch
MR-2535-failures-v1.patch should apply to both 0.20.205 and security-trunk
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch, MR-2535-failures-v1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when readFields() called
[
https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045107#comment-13045107
]
Todd Lipcon commented on MAPREDUCE-2557:
Hi William. Thanks for the bug report. Two things:
- a silly nit: can you use "1L" and "2L" instead of "1l" and "2l" in the tests?
I was staring at it for 5 minutes before I realized those were 'l' ("ell") and
not '1' ("one").
- what ramifications, if any, does this have on the MR framework itself? Would
this cause incorrect counters in any usual use case? How'd you come across the
bug?
> Counters don't reset state when readFields() called
> ---
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: William Slacum
>Priority: Trivial
> Attachments: MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not
> completely reset. The IdentityHashMap, Counter> cache retains all
> previous mappings, even after the actual CounterGroups are changed. Using the
> same Counters pointer over and over again results in the cache always keeping
> the mapping for the first call to getCounter(Enum). I've add a clear()
> call to the cache when readFields() is called and added a unit test to verify
> that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile
[
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reopened MAPREDUCE-2535:
The fix is good, but it broke the system tests. Reopening the bug to add in a
patch to fix the tests.
> JobClient creates a RunningJob with null status and profile
> ---
>
> Key: MAPREDUCE-2535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 0.20.204.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-2535-0.20.20X-V1.patch
>
>
> Exception occurred because the job was retired and is removed from
> RetireJobCcahe and CompletedJobStatusStore. But, the
> JobClient creates a RunningJob with null status and profile, if getJob(JobID)
> is called again.
> So, Even-though not null check is there in the following user code, it did
> not help.
> 466 runningJob = jobClient.getJob(mapRedJobID);
> 467 if(runningJob != null) {
> JobClient.getJob() should return null if status is null.
> In trunk this is fixed by validating that the job status is not null every
> time it is updated, and also verifying that that the profile data is not null
> when created.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2512) wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
[
https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated MAPREDUCE-2512:
---
Status: Open (was: Patch Available)
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000)
> in reduce task when there are no copies in progress and no new copies to
> schedule
> -
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 0.20.2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
>try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is
> going to wait for 5000 millis. Instead of waiting for 5000 millis, this
> thread can wait with timeout and GetMapEventsThread can notify it if gets new
> map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2549) Potential resource leaks in HadoopServer.java, RunOnHadoopWizard.java and Environment.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045098#comment-13045098 ] Todd Lipcon commented on MAPREDUCE-2549: Since this is part of the eclipse plugin which doesn't have unit tests, it's OK not to have them. But: have you built and used the eclipse plugin with this patch applied? ie manually tested? > Potential resource leaks in HadoopServer.java, RunOnHadoopWizard.java and > Environment.java > -- > > Key: MAPREDUCE-2549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/eclipse-plugin, contrib/streaming >Affects Versions: 0.23.0 >Reporter: Devaraj K >Assignee: Devaraj K > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2549.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
[
https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045101#comment-13045101
]
Todd Lipcon commented on MAPREDUCE-2512:
As this is an optimization, you should prepare a patch against trunk if the
issue exists there. We don't generally put optimizations backward into prior
releases.
> wait(5000) and notify() mechanism can be implemented instead of sleep(5000)
> in reduce task when there are no copies in progress and no new copies to
> schedule
> -
>
> Key: MAPREDUCE-2512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 0.20.2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.20.4
>
> Attachments: MAPREDUCE-2512.patch
>
>
> {code:title=ReduceTask.java|borderStyle=solid}
>try {
> if (numInFlight == 0 && numScheduled == 0) {
> // we should indicate progress as we don't want TT to think
> // we're stuck and kill us
> reporter.progress();
> Thread.sleep(5000);
> }
> } catch (InterruptedException e) { } // IGNORE
> {code}
> Here if we have no copies in flight and we can't schedule anything new, it is
> going to wait for 5000 millis. Instead of waiting for 5000 millis, this
> thread can wait with timeout and GetMapEventsThread can notify it if gets new
> map completion events earlier than 5000 millis time.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2378) Reduce fails when running on 1 small file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Baff updated MAPREDUCE-2378: -- Attachment: failed reduce task log.html Log for a failed Reduce task that exhibits this behavior. In looking at the thread states of the remaining threads, it appears that the ShuffleSort has hit a deadlock, where for some reason the InterruptedIOException wasn't caught and so a Fetcher thread failed to complete normally. Not really sure why this might have occured, I'm not terribly familiar with the code base, although I'm wondering ifThis issue from log4j could be related https://issues.apache.org/bugzilla/show_bug.cgi?id=44157. This is fixed in 1.2.16 released on April 2010. Hadoop is still using 1.2.15. > Reduce fails when running on 1 small file. > --- > > Key: MAPREDUCE-2378 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2378 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 > Environment: java version "1.6.0_07" > Diablo Java(TM) SE Runtime Environment (build 1.6.0_07-b02) > Diablo Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) >Reporter: Simon Dircks > Labels: 1, failed, file, log4j, reduce, single, small, tiny > Attachments: failed reduce task log.html > > > If i run the wordcount example on 1 small (less than 2MB) file i get the > following error: > log4j:ERROR Failed to flush writer, > java.io.InterruptedIOException > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) > at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) > at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276) > at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) > at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) > at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58) > at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:316) > at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) > at > org.apache.hadoop.mapred.TaskLogAppender.append(TaskLogAppender.java:58) > at > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) > at > org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) > at org.apache.log4j.Category.callAppenders(Category.java:206) > at org.apache.log4j.Category.forcedLog(Category.java:391) > at org.apache.log4j.Category.log(Category.java:856) > at > org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:199) > at > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.freeHost(ShuffleScheduler.java:345) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152) > If i run the wordcount test with 2 files, it works fine. > I have actually repeated this with my own code. I am working on something > that requires me to map/reduce a small file and I had to work around the > problem by splitting the file into 2 1MB pieces for my job to run. > All our jobs that run on 1 single larger file (over 1GB) work flawlessly. I > am not exactly sure the threshold, From the testing i have done it seems to > be any file smaller than the default HDFS block size (64MB) Sometimes it > seems random in the 5-64MB range. But its 100% for the 5MB and smaller files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2570) Bug in RAID FS (DistributedRaidFileSystem) unraid path
Bug in RAID FS (DistributedRaidFileSystem) unraid path -- Key: MAPREDUCE-2570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2570 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid Reporter: Ramkumar Vadali Assignee: Ramkumar Vadali The "un-raid" path in DistributedRaidFileSystem goes through RaidNode.unRaidCorruptBlock(), which has a bug when the parity file is inside a HAR. The temporary file that contains the recovered block contents is created in the filesystem that hosts the parity file. In case the parity file is inside a HAR, its filesystem is HarFileSystem, which is read-only. In this case the temporary file creation will fail. The fix is a one-line change to use the underlying filesystem of the HAR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Naisbitt updated MAPREDUCE-2489: Attachment: MAPREDUCE-2489-mapred-v2.patch Address the comments Robert Evans made in HADOOP-7314, making urlValidator a static member variable. > Jobsplits with random hostnames can make the queue unusable > --- > > Key: MAPREDUCE-2489 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Jeffrey Naisbitt >Assignee: Jeffrey Naisbitt > Attachments: MAPREDUCE-2489-mapred-v2.patch, > MAPREDUCE-2489-mapred.patch > > > We saw an issue where a custom InputSplit was returning invalid hostnames for > the splits that were then causing the JobTracker to attempt to excessively > resolve host names. This caused a major slowdown for the JobTracker. We > should prevent invalid InputSplit hostnames from affecting everyone else. > I propose we implement some verification for the hostnames to try to ensure > that we only do DNS lookups on valid hostnames (and fail otherwise). We > could also fail the job after a certain number of failures in the resolve. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2568) Fix TestFileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045030#comment-13045030 ] Jeffrey Naisbitt commented on MAPREDUCE-2568: - +1 Looks good to me. Should we have a negative test that verifies the original testcase fails here, or do you already have that tested elsewhere? > Fix TestFileSystem > -- > > Key: MAPREDUCE-2568 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2568 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: MAPREDUCE-2568.patch > > > CommandFormat which is used to parse command lines is following posix > conventions of stopping at the first non-argument. There is just one test in > this file which placed an option in the middle of the args. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2569) MR-279: Restarting resource manager with root capacity not equal to 100 percent should result in error
[ https://issues.apache.org/jira/browse/MAPREDUCE-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2569: --- Description: root.capacity is set to 90% without failure > MR-279: Restarting resource manager with root capacity not equal to 100 > percent should result in error > -- > > Key: MAPREDUCE-2569 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2569 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > > root.capacity is set to 90% without failure -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2569) MR-279: Restarting resource manager with root capacity not equal to 100 percent should result in error
MR-279: Restarting resource manager with root capacity not equal to 100 percent should result in error -- Key: MAPREDUCE-2569 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2569 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2487) ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
[ https://issues.apache.org/jira/browse/MAPREDUCE-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045024#comment-13045024 ] Hudson commented on MAPREDUCE-2487: --- Integrated in Hadoop-Mapreduce-trunk-Commit #716 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/716/]) MAPREDUCE-2487. ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE. Contributed by Devaraj K. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132719 Files : * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/lib/Chain.java * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/lib/TestChain.java > ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE > - > > Key: MAPREDUCE-2487 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2487 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Forrest Vines >Assignee: Devaraj K >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2487.patch > > > On line 293 of o.a.h.mapred.lib.Chain in setReducer(...): > reducerConf.setBoolean(MAPPER_BY_VALUE, byValue); > this should be REDUCER_BY_VALUE. > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2568) Fix TestFileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-2568: --- Status: Patch Available (was: Open) > Fix TestFileSystem > -- > > Key: MAPREDUCE-2568 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2568 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: MAPREDUCE-2568.patch > > > CommandFormat which is used to parse command lines is following posix > conventions of stopping at the first non-argument. There is just one test in > this file which placed an option in the middle of the args. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2487) ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
[ https://issues.apache.org/jira/browse/MAPREDUCE-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2487: --- Resolution: Fixed Fix Version/s: (was: 0.23.0) 0.22.0 Assignee: Devaraj K Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to 22 and trunk. Thanks Devaraj and Forrest. > ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE > - > > Key: MAPREDUCE-2487 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2487 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Forrest Vines >Assignee: Devaraj K >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2487.patch > > > On line 293 of o.a.h.mapred.lib.Chain in setReducer(...): > reducerConf.setBoolean(MAPPER_BY_VALUE, byValue); > this should be REDUCER_BY_VALUE. > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2568) Fix TestFileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-2568: --- Attachment: MAPREDUCE-2568.patch Move option earlier. > Fix TestFileSystem > -- > > Key: MAPREDUCE-2568 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2568 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: MAPREDUCE-2568.patch > > > CommandFormat which is used to parse command lines is following posix > conventions of stopping at the first non-argument. There is just one test in > this file which placed an option in the middle of the args. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2568) Fix TestFileSystem
Fix TestFileSystem -- Key: MAPREDUCE-2568 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2568 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp CommandFormat which is used to parse command lines is following posix conventions of stopping at the first non-argument. There is just one test in this file which placed an option in the middle of the args. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2487) ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
[ https://issues.apache.org/jira/browse/MAPREDUCE-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044948#comment-13044948 ] Hadoop QA commented on MAPREDUCE-2487: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481568/MAPREDUCE-2487.patch against trunk revision 1132529. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/350//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/350//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/350//console This message is automatically generated. > ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE > - > > Key: MAPREDUCE-2487 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2487 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Forrest Vines >Priority: Minor > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2487.patch > > > On line 293 of o.a.h.mapred.lib.Chain in setReducer(...): > reducerConf.setBoolean(MAPPER_BY_VALUE, byValue); > this should be REDUCER_BY_VALUE. > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-143) OOM in the TaskTracker while serving map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044932#comment-13044932 ] Sateesh Lakkarsu commented on MAPREDUCE-143: I think we ran into the same issue, any work around or config tweak to avoid running into this? Thanks. > OOM in the TaskTracker while serving map outputs > > > Key: MAPREDUCE-143 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-143 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Devaraj Das > > Saw this exception in the TT logs: > 2009-02-06 06:18:08,553 ERROR org.mortbay.log: EXCEPTION > java.lang.OutOfMemoryError: GC overhead limit exceeded > 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput > java.lang.OutOfMemoryError: GC overhead limit exceeded > 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39) > at java.nio.ByteBuffer.allocate(ByteBuffer.java:312) > at > org.mortbay.io.nio.IndirectNIOBuffer.(IndirectNIOBuffer.java:28) > at > org.mortbay.jetty.nio.AbstractNIOConnector.newBuffer(AbstractNIOConnector.java:71) > at > org.mortbay.jetty.AbstractBuffers.getBuffer(AbstractBuffers.java:131) > at org.mortbay.jetty.HttpGenerator.addContent(HttpGenerator.java:145) > at > org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:642) > at > org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577) > at > org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2879) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2487) ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
[ https://issues.apache.org/jira/browse/MAPREDUCE-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2487: - Fix Version/s: 0.23.0 Status: Patch Available (was: Open) > ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE > - > > Key: MAPREDUCE-2487 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2487 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Forrest Vines >Priority: Minor > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2487.patch > > > On line 293 of o.a.h.mapred.lib.Chain in setReducer(...): > reducerConf.setBoolean(MAPPER_BY_VALUE, byValue); > this should be REDUCER_BY_VALUE. > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2487) ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
[ https://issues.apache.org/jira/browse/MAPREDUCE-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2487: - Attachment: MAPREDUCE-2487.patch > ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE > - > > Key: MAPREDUCE-2487 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2487 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Forrest Vines >Priority: Minor > Attachments: MAPREDUCE-2487.patch > > > On line 293 of o.a.h.mapred.lib.Chain in setReducer(...): > reducerConf.setBoolean(MAPPER_BY_VALUE, byValue); > this should be REDUCER_BY_VALUE. > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
