[jira] [Resolved] (MAPREDUCE-3410) Capacity scheduler reconfiguration of queues does not work for add sub-queues to an existing queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3410. -- Resolution: Later For now we should punt on this - this is akin to 'delete queue' as Jason explained. > Capacity scheduler reconfiguration of queues does not work for add sub-queues > to an existing queue > -- > > Key: MAPREDUCE-3410 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3410 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Anupam Seth > > If we have an existing queue configuration such as > root > |---> A > |---> B > and we attempt to reconfigure it so that we now have > root > |---> A >|---> A1 >|---> A2 > |---> B > we get an IOException as follows: > java.io.IOException: Failed to re-init queues > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:197) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.testInitializeQueue(TestLeafQueue.java:206) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at > org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) > at org.junit.runners.ParentRunner.run(ParentRunner.java:236) > at > org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:45) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) > at > org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:172) > at > org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:78) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:70) > Caused by: java.io.IOException: Trying to reinitialize root.a from root.a > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.reinitialize(LeafQueue.java:524) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.reinitialize(ParentQueue.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:240) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:194) > ... 32 more > This is apparently because the CapacityScheduler still wants to think of A as > a LeafQueue instead of realizing it to be updated as a ParentQueue. > Maybe, this use case is not supposed to be supported, in which case,
[jira] [Created] (MAPREDUCE-4524) Capacity Scheduler does not support adding sub-queues to the existing queues.
Kiran BC created MAPREDUCE-4524: --- Summary: Capacity Scheduler does not support adding sub-queues to the existing queues. Key: MAPREDUCE-4524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4524 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched, documentation Affects Versions: 0.23.0 Reporter: Kiran BC In-line to the issue, MAPREDUCE-3410, there should be a note stating that - "Capacity Scheduler does not support adding sub-queues to the existing queue" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-806) WordCount example does not compile given the current instructions
[ https://issues.apache.org/jira/browse/MAPREDUCE-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430787#comment-13430787 ] sakurai commented on MAPREDUCE-806: --- sorry, i missed out the jar command > WordCount example does not compile given the current instructions > - > > Key: MAPREDUCE-806 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-806 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 0.20.1 > Environment: linux >Reporter: Hector Yuen >Priority: Trivial > > http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Example%3A+WordCount+v1.0 > In this example, the classpath is missing commons-cli-2.0-SNAPSHOT.jar > If we compile according to the instructions: > $ javac -classpath /hadoop/core/hadoop-0.20.0-core.jar -d ioperf_classes/ > src/WordCount.java > src/WordCount.java:54: cannot access org.apache.commons.cli.Options > class file for org.apache.commons.cli.Options not found > String[] otherArgs = new GenericOptionsParser(conf, > args).getRemainingArgs(); > The correct compilation should be > $ javac -classpath > /hadoop/core/hadoop-0.20.0-core.jar:/hadoop/core/lib/commons-cli-2.0-SNAPSHOT.jar > -d ioperf_classes/ src/WordCount.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4511) Add IFile readahead
[ https://issues.apache.org/jira/browse/MAPREDUCE-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430744#comment-13430744 ] Hadoop QA commented on MAPREDUCE-4511: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539743/MAPREDUCE-4511_branch-1_rev2.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2717//console This message is automatically generated. > Add IFile readahead > --- > > Key: MAPREDUCE-4511 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4511 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-4511_branch-1_rev2.patch, > MAPREDUCE-4511_branch1.patch, MAPREDUCE-4511_trunk.patch, > MAPREDUCE-4511_trunk_rev2.patch > > > This ticket is to add IFile readahead as part of HADOOP-7714. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4511) Add IFile readahead
[ https://issues.apache.org/jira/browse/MAPREDUCE-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-4511: Attachment: MAPREDUCE-4511_branch-1_rev2.patch MAPREDUCE-4511_trunk_rev2.patch > Add IFile readahead > --- > > Key: MAPREDUCE-4511 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4511 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-4511_branch-1_rev2.patch, > MAPREDUCE-4511_branch1.patch, MAPREDUCE-4511_trunk.patch, > MAPREDUCE-4511_trunk_rev2.patch > > > This ticket is to add IFile readahead as part of HADOOP-7714. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4511) Add IFile readahead
[ https://issues.apache.org/jira/browse/MAPREDUCE-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430732#comment-13430732 ] Ahmed Radwan commented on MAPREDUCE-4511: - Here are updated versions for both trunk and branch-1 adding the ability to disable and configure the readahead and its length, respectively. > Add IFile readahead > --- > > Key: MAPREDUCE-4511 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4511 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-4511_branch1.patch, MAPREDUCE-4511_trunk.patch > > > This ticket is to add IFile readahead as part of HADOOP-7714. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4044) YarnClientProtocolProvider does not honor mapred.job.tracker property
[ https://issues.apache.org/jira/browse/MAPREDUCE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430697#comment-13430697 ] Mohammad Kamrul Islam commented on MAPREDUCE-4044: -- Is there any update on this? > YarnClientProtocolProvider does not honor mapred.job.tracker property > - > > Key: MAPREDUCE-4044 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4044 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Alejandro Abdelnur > > The YarnClientProtocolProvider/YARNRunner/ResourceMgrDelegate bootstrap only > looks for 'yarn.resourcemanager.address', they ignore 'mapred.job.tracker' > This breaks backward compatibility and creates issues in Oozie. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4323) NM log aggregator leaks filesystems
[
https://issues.apache.org/jira/browse/MAPREDUCE-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430695#comment-13430695
]
Jonathan Eagles commented on MAPREDUCE-4323:
Thanks to [~jlowe] for pointing out a simple way for me to check the socket
leak.
{code}
while true; do ls /proc//fd/ | wc; sleep 1; done
{code}
> NM log aggregator leaks filesystems
> ---
>
> Key: MAPREDUCE-4323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4323
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: MAPREDUCE-4323.patch
>
>
> The NM is exhausting its fds because it's not closing fs instances when the
> app is finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4323) NM log aggregator leaks filesystems
[ https://issues.apache.org/jira/browse/MAPREDUCE-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430692#comment-13430692 ] Jonathan Eagles commented on MAPREDUCE-4323: This patch seems like it partially addresses the problem. Something else still seems to be leaking sockets. > NM log aggregator leaks filesystems > --- > > Key: MAPREDUCE-4323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4323 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: MAPREDUCE-4323.patch > > > The NM is exhausting its fds because it's not closing fs instances when the > app is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[ https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430658#comment-13430658 ] Daryn Sharp commented on MAPREDUCE-3782: +1 I'm not fond of how formatters have to explicitly obtain tokens, but this fits the current paradigm. Thanks Jason! > teragen terasort jobs fail when using webhdfs:// > - > > Key: MAPREDUCE-3782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3782.patch > > > When running a teragen job with a webhdfs:// url the delegation token that is > retrieved is an hdfs delegation token. > And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4039) Sort Avoidance
[
https://issues.apache.org/jira/browse/MAPREDUCE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430587#comment-13430587
]
Mariappan Asokan commented on MAPREDUCE-4039:
-
Hi Anty,
I have submitted a patch for MAPREDUCE-2454 to support a pluggable sort for
MR. I have some ideas on implementing a NullSortPlugin which will be a special
sort plugin that avoids sorting. The NullSortPlugin can live outside the
Hadoop MR code. I can share my thoughts if you are interested.
Thanks.
-- Asokan
> Sort Avoidance
> --
>
> Key: MAPREDUCE-4039
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4039
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
>Affects Versions: 0.23.2
>Reporter: anty.rao
>Assignee: anty
>Priority: Minor
> Fix For: 0.23.2
>
> Attachments: IndexedCountingSortable.java,
> MAPREDUCE-4039-branch-0.23.2.patch, MAPREDUCE-4039-branch-0.23.2.patch,
> MAPREDUCE-4039-branch-0.23.2.patch
>
>
> Inspired by
> [Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
> in 5.1 MapReduce Enhanceemtns:
> {quote}*Sort Avoidance*. Certain operators such as hash join
> and hash aggregation require shuffling, but not sorting. The
> MapReduce API was enhanced to automatically turn off
> sorting for these operations. When sorting is turned off, the
> mapper feeds data to the reducer which directly passes the
> data to the Reduce() function bypassing the intermediate
> sorting step. This makes many SQL operators significantly
> more ecient.{quote}
> There are a lot of applications which need aggregation only, not
> sorting.Using sorting to achieve aggregation is costly and inefficient.
> Without sorting, up application can make use of hash table or hash map to do
> aggregation efficiently.But application should bear in mind that reduce
> memory is limited, itself is committed to manage memory of reduce, guard
> against out of memory. Map-side combiner is not supported, you can also do
> hash aggregation in map side as a workaround.
> the following is the main points of sort avoidance implementation
> # add a configuration parameter ??mapreduce.sort.avoidance??, boolean type,
> to turn on/off sort avoidance workflow.Two type of workflow are coexist
> together.
> # key/value pairs emitted by map function is sorted by partition only, using
> a more efficient sorting algorithm: counting sort.
> # map-side merge, use a kind of byte merge, which just concatenate bytes from
> generated spills, read in bytes, write out bytes, without overhead of
> key/value serialization/deserailization, comparison, which current version
> incurs.
> # reduce can start up as soon as there is any map output available, in
> contrast to sort workflow which must wait until all map outputs are fetched
> and merged.
> # map output in memory can be directly consumed by reduce.When reduce can't
> catch up with the speed of incoming map outputs, in-memory merge thread will
> kick in, merging in-memory map outputs onto disk.
> # sequentially read in on-disk files to feed reduce, in contrast to currently
> implementation which read multiple files concurrently, result in many disk
> seek. Map output in memory take precedence over on disk files in feeding
> reduce function.
> I have already implement this feature based on hadoop CDH3U3 and done some
> performance evaluation, you can reference to
> [https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
> into yarn. Welcome for commenting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[ https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430583#comment-13430583 ] Hadoop QA commented on MAPREDUCE-3782: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539702/MAPREDUCE-3782.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-examples. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2716//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2716//console This message is automatically generated. > teragen terasort jobs fail when using webhdfs:// > - > > Key: MAPREDUCE-3782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3782.patch > > > When running a teragen job with a webhdfs:// url the delegation token that is > retrieved is an hdfs delegation token. > And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4323) NM log aggregator leaks filesystems
[ https://issues.apache.org/jira/browse/MAPREDUCE-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4323: --- Summary: NM log aggregator leaks filesystems (was: NM leaks sockets) > NM log aggregator leaks filesystems > --- > > Key: MAPREDUCE-4323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4323 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: MAPREDUCE-4323.patch > > > The NM is exhausting its fds because it's not closing fs instances when the > app is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[ https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3782: -- Assignee: Jason Lowe Target Version/s: 0.23.3, 2.2.0-alpha Status: Patch Available (was: Open) > teragen terasort jobs fail when using webhdfs:// > - > > Key: MAPREDUCE-3782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3782.patch > > > When running a teragen job with a webhdfs:// url the delegation token that is > retrieved is an hdfs delegation token. > And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[ https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3782: -- Attachment: MAPREDUCE-3782.patch Patch for TeraOutputFormat delegation token fix. No automated testcase, but I did manually test on a secure cluster and verified that teragen obtains the delegation token for webhdfs. > teragen terasort jobs fail when using webhdfs:// > - > > Key: MAPREDUCE-3782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Priority: Critical > Attachments: MAPREDUCE-3782.patch > > > When running a teragen job with a webhdfs:// url the delegation token that is > retrieved is an hdfs delegation token. > And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[
https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430564#comment-13430564
]
Jason Lowe commented on MAPREDUCE-3782:
---
Took a closer look at this. The log message about getting a delegation token
for HDFS is not in error, as that token is needed to setup the staging
directory for the job. So that is working as-expected.
Teragen failing to get a delegation token for webhdfs *is* a bug in
TeraOutputFormat, as it is responsible for grabbing any necessary delegation
tokens in the {{checkOutputSpecs}} method.
Terasort grabs two tokens, one from webhdfs (for the input files and would also
for the output directory once TeraOutputFormat is fixed) and one from HDFS for
the job staging area. We won't see two delegation token messages for the same
filesystem since it doesn't grab a new token if it already has one.
The range error is a known problem that has been already fixed, see HDFS-3101.
So to sum up, I think there's only one remaining bug here: TeraOutputFormat is
not grabbing a delegation token for the output directory when
{{checkOutputSpecs}} is called.
> teragen terasort jobs fail when using webhdfs://
> -
>
> Key: MAPREDUCE-3782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Arpit Gupta
>Priority: Critical
>
> When running a teragen job with a webhdfs:// url the delegation token that is
> retrieved is an hdfs delegation token.
> And the subsequent terasort job on the output fails with java io exception
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4323) NM leaks sockets
[ https://issues.apache.org/jira/browse/MAPREDUCE-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430560#comment-13430560 ] Hadoop QA commented on MAPREDUCE-4323: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539698/MAPREDUCE-4323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2715//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2715//console This message is automatically generated. > NM leaks sockets > > > Key: MAPREDUCE-4323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4323 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: MAPREDUCE-4323.patch > > > The NM is exhausting its fds because it's not closing fs instances when the > app is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4323) NM leaks sockets
[
https://issues.apache.org/jira/browse/MAPREDUCE-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4323:
---
Attachment: MAPREDUCE-4323.patch
Patch ensures {{closeAllForUGI}} is called in the event of a service init
failure, or when the aggregation thread is done.
> NM leaks sockets
>
>
> Key: MAPREDUCE-4323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4323
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: MAPREDUCE-4323.patch
>
>
> The NM is exhausting its fds because it's not closing fs instances when the
> app is finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4323) NM leaks sockets
[ https://issues.apache.org/jira/browse/MAPREDUCE-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4323: --- Status: Patch Available (was: Open) > NM leaks sockets > > > Key: MAPREDUCE-4323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4323 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.0.0-alpha, 0.23.0, 0.24.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: MAPREDUCE-4323.patch > > > The NM is exhausting its fds because it's not closing fs instances when the > app is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4498) Remove hsqldb jar from Hadoop runtime classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430334#comment-13430334 ] Hudson commented on MAPREDUCE-4498: --- Integrated in Hadoop-Hdfs-trunk #1128 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1128/]) MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml > Remove hsqldb jar from Hadoop runtime classpath > --- > > Key: MAPREDUCE-4498 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4498 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, examples >Affects Versions: 2.2.0-alpha >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Critical > Fix For: 2.2.0-alpha > > Attachments: MAPREDUCE-4498-v2.patch, MAPREDUCE-4498-v3.patch, > MAPREDUCE-4498.patch > > > The hsqldb jar is included in hadoop for the DBCountPageView example only. > Currently the example is using hsqldb version 2.x; however, 2.x is > incompatible with 1.8.x -- having this jar in the hadoop class path conflicts > with dependent projects like Oozie, Hive, and Pig which still use 1.8.x. As > there are no features hsqldb 2.x that are used by the example, we should > remove it from Hadoop's runtime classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4484) Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration
[ https://issues.apache.org/jira/browse/MAPREDUCE-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430331#comment-13430331 ] Hudson commented on MAPREDUCE-4484: --- Integrated in Hadoop-Hdfs-trunk #1128 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1128/]) MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration. (ahmed.radwan via tucu) (Revision 1370120) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370120 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java > Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration > - > > Key: MAPREDUCE-4484 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4484 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Fix For: 2.2.0-alpha > > Attachments: MAPREDUCE-4484.patch > > > Noticed that the IS_MINI_YARN_CLUSTER property name in YarnConfiguration > ended up having an extra "." after appending to YARN_PREFIX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4494) TestFifoScheduler failing with Metrics source QueueMetrics,q0=default already exists!
[
https://issues.apache.org/jira/browse/MAPREDUCE-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430332#comment-13430332
]
Hudson commented on MAPREDUCE-4494:
---
Integrated in Hadoop-Hdfs-trunk #1128 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1128/])
MAPREDUCE-4494. TestFifoScheduler failing with Metrics source
QueueMetrics,q0=default already exists. (ahmed.radwan via tucu) (Revision
1370118)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370118
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> TestFifoScheduler failing with Metrics source QueueMetrics,q0=default already
> exists!
> -
>
> Key: MAPREDUCE-4494
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4494
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.0-alpha
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Fix For: 2.2.0-alpha
>
> Attachments: MAPREDUCE-4494.patch
>
>
> TestFifoScheduler is failing:
> {code}
> Running org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.614 sec
> <<< FAILURE!
> Results :
> Tests in error:
> test(org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler):
> Metrics source QueueMetrics,q0=default already exists!
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-806) WordCount example does not compile given the current instructions
[ https://issues.apache.org/jira/browse/MAPREDUCE-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429987#comment-13429987 ] sakurai commented on MAPREDUCE-806: --- i came across here as i happened to be facing the same problem. everything goes well after i followed the instruction here(no errors after this), but i cant see any wordcount.jar created.. javac -classpath /home/hadoop/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/hadoop/lib/commons-cli-1.2.jar -d classes src/WordCount.java i only have the following classes created. -rw-rw-r-- 1 hadoop hadoop 1790 Aug 7 15:42 WordCount$TokenizerMapper.class -rw-rw-r-- 1 hadoop hadoop 1789 Aug 7 15:42 WordCount$IntSumReducer.class -rw-rw-r-- 1 hadoop hadoop 1911 Aug 7 15:42 WordCount.class sorry, i am not familiar with java, can someone tell me if my compiling is correct? thank you. > WordCount example does not compile given the current instructions > - > > Key: MAPREDUCE-806 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-806 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 0.20.1 > Environment: linux >Reporter: Hector Yuen >Priority: Trivial > > http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Example%3A+WordCount+v1.0 > In this example, the classpath is missing commons-cli-2.0-SNAPSHOT.jar > If we compile according to the instructions: > $ javac -classpath /hadoop/core/hadoop-0.20.0-core.jar -d ioperf_classes/ > src/WordCount.java > src/WordCount.java:54: cannot access org.apache.commons.cli.Options > class file for org.apache.commons.cli.Options not found > String[] otherArgs = new GenericOptionsParser(conf, > args).getRemainingArgs(); > The correct compilation should be > $ javac -classpath > /hadoop/core/hadoop-0.20.0-core.jar:/hadoop/core/lib/commons-cli-2.0-SNAPSHOT.jar > -d ioperf_classes/ src/WordCount.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
