[jira] [Assigned] (MAPREDUCE-4039) Sort Avoidance
[
https://issues.apache.org/jira/browse/MAPREDUCE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-4039:
Assignee: anty
Sort Avoidance
--
Key: MAPREDUCE-4039
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4039
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 0.23.2
Reporter: anty.rao
Assignee: anty
Priority: Minor
Fix For: 0.23.2
Attachments: MAPREDUCE-4039-branch-0.23.2.patch,
MAPREDUCE-4039-branch-0.23.2.patch
Inspired by
[Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
in 5.1 MapReduce Enhanceemtns:
{quote}*Sort Avoidance*. Certain operators such as hash join
and hash aggregation require shuffling, but not sorting. The
MapReduce API was enhanced to automatically turn off
sorting for these operations. When sorting is turned off, the
mapper feeds data to the reducer which directly passes the
data to the Reduce() function bypassing the intermediate
sorting step. This makes many SQL operators significantly
more ecient.{quote}
There are a lot of applications which need aggregation only, not
sorting.Using sorting to achieve aggregation is costly and inefficient.
Without sorting, up application can make use of hash table or hash map to do
aggregation efficiently.But application should bear in mind that reduce
memory is limited, itself is committed to manage memory of reduce, guard
against out of memory. Map-side combiner is not supported, you can also do
hash aggregation in map side as a workaround.
the following is the main points of sort avoidance implementation
# add a configuration parameter ??mapreduce.sort.avoidance??, boolean type,
to turn on/off sort avoidance workflow.Two type of workflow are coexist
together.
# key/value pairs emitted by map function is sorted by partition only, using
a more efficient sorting algorithm: counting sort.
# map-side merge, use a kind of byte merge, which just concatenate bytes from
generated spills, read in bytes, write out bytes, without overhead of
key/value serialization/deserailization, comparison, which current version
incurs.
# reduce can start up as soon as there is any map output available, in
contrast to sort workflow which must wait until all map outputs are fetched
and merged.
# map output in memory can be directly consumed by reduce.When reduce can't
catch up with the speed of incoming map outputs, in-memory merge thread will
kick in, merging in-memory map outputs onto disk.
# sequentially read in on-disk files to feed reduce, in contrast to currently
implementation which read multiple files concurrently, result in many disk
seek. Map output in memory take precedence over on disk files in feeding
reduce function.
I have already implement this feature based on hadoop CDH3U3 and done some
performance evaluation, you can reference to
[https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
into yarn. Welcome for commenting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-4025:
Assignee: Jason Lowe
AM can crash if task attempt reports bogus progress value
-
Key: MAPREDUCE-4025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
Attachments: MAPREDUCE-4025.patch
If a task attempt reports a bogus progress value (e.g.: something above 1.0)
then the AM can crash like this:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 12
at
org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
at
org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3921: Assignee: Bikas Saha MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy Key: MAPREDUCE-3921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3833) Capacity scheduler queue refresh doesn't recompute queue capacities properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3833: Assignee: Jason Lowe (was: Arun C Murthy) Jason deserves credit for the hardwork here... Capacity scheduler queue refresh doesn't recompute queue capacities properly Key: MAPREDUCE-3833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3833 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-3833-testcase.patch, MAPREDUCE-3833.patch Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3353: Assignee: Vinod Kumar Vavilapalli Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Critical Fix For: 0.23.1 When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3747) Memory Total is not refreshed until an app is launched
[ https://issues.apache.org/jira/browse/MAPREDUCE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3747: Assignee: Arun C Murthy Memory Total is not refreshed until an app is launched -- Key: MAPREDUCE-3747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3747 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.1 Memory Total on the RM UI is not refreshed until an application is launched. This is a problem when the cluster is started for the first time or when there are any lost/decommissioned NMs. When the cluster is started for the first time, Active Nodes is 0 but the Memory Total=0. Also when there are any lost/decommissioned nodes, Memory Total has wrong value. This is a useful tool for cluster admins and has to be updated correctly without having the need to submit an app each time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3784) maxActiveApplications(|PerUser) per queue is too low for small clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3784: Assignee: Arun C Murthy maxActiveApplications(|PerUser) per queue is too low for small clusters --- Key: MAPREDUCE-3784 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3784 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy We ran into this issue while testing on small clusters. On a 7node cluster with 8G per node, for a queue with absolute capacity 30%, user limit 100%, maxActiveApplications and maxActiveApplicationsPerUser is calculated to be 1. This means that even though the queue has 17GB(0.3*8*7), only 1 user can run 1 app at a given time queuing up rest of the apps/users. This hurts performance on small clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3427:
Assignee: Hitesh Shah
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3640: Assignee: Arun C Murthy (was: Siddharth Seth) AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3763) Failed refreshQueues due to misconfiguration prevents further refreshing of queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3763: Assignee: Arun C Murthy Failed refreshQueues due to misconfiguration prevents further refreshing of queues -- Key: MAPREDUCE-3763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3763 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.1 Stumbled upon this problem while refreshing queues with incorrect configuration. The exact scenario was: 1. Added a new queue newQueue without defining its capacity. 2. bin/mapred queue -refreshQueues fails correctly with Illegal capacity of -1 for queue root.newQueue 3. However, after defining the capacity of newQueue followed by a second bin/mapred queue -refreshQueues throws org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=newQueue already exists! Also see Hadoop:name=QueueMetrics,q0=root,q1=newQueue,service=ResourceManager metrics being available even though the queue was not added. The expected behavior would be to refresh the queues correctly and allow addition of newQueue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3699) Default RPC handlers are very low for YARN servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3699: Assignee: Hitesh Shah Default RPC handlers are very low for YARN servers -- Key: MAPREDUCE-3699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Hitesh Shah Fix For: 0.23.1 Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3681: Assignee: Arun C Murthy capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3683) Capacity scheduler LeafQueues maximum capacity calculation issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3683: Assignee: Arun C Murthy Capacity scheduler LeafQueues maximum capacity calculation issues - Key: MAPREDUCE-3683 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity. It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3612) Task.TaskReporter.done method blocked for some time when task is finishing
[
https://issues.apache.org/jira/browse/MAPREDUCE-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3612:
Assignee: Binglin Chang
Task.TaskReporter.done method blocked for some time when task is finishing
--
Key: MAPREDUCE-3612
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3612
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: MAPREDUCE-3612.patch
We recently have done some tests to evaluate performances of different Hadoop
versions(1.0, 0.23, Baidu internal version), and found some weird results.
One of them is in 1.0 Task.TaskReporter.done() takes too much time, about 2s,
this is bad for small tasks. After reviewing source code and add some log,
the following code block Task.TaskReporter.done
{code:title=src/mapred/org/apache/hadoop/mapred/Task.java}
658 try {
659 Thread.sleep(PROGRESS_INTERVAL);
660 }
723 public void stopCommunicationThread() throws InterruptedException {
724 // Updating resources specified in ResourceCalculatorPlugin
725 if (pingThread != null) {
726 synchronized(lock) {
727 while(!done) {
728 lock.wait();
729 }
730 }
731 pingThread.interrupt();
732 pingThread.join();
733 }
734 }
{code}
Originally line 724-730 don't exists, I don't know why it is added. If it is
needed, we can replace Thread.sleep with Object.wait(timeout) and
Object.notify instead, so it won't block.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3596:
Assignee: Vinod Kumar Vavilapalli
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Attachments: logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3605) Allow mr commands to be run via bin/hadoop
[
https://issues.apache.org/jira/browse/MAPREDUCE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3605:
Assignee: Arun C Murthy
Allow mr commands to be run via bin/hadoop
--
Key: MAPREDUCE-3605
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3605
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Assignee: Arun C Murthy
Fix For: 0.23.1
MR command line options are not supported in bin/hadoop.
{noformat}
bin/hadoop job
Exception in thread main java.lang.NoClassDefFoundError: job
Caused by: java.lang.ClassNotFoundException: job
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: job. Program will exit.
{noformat}
A deprecated message like DEPRECATED: Use of this script to execute mapred
command is deprecated. Instead use the mapred command for it. should be
displayed along with the correct output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3490) RMContainerAllocator counts failed maps towards Reduce ramp up
[ https://issues.apache.org/jira/browse/MAPREDUCE-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3490: Assignee: Arun C Murthy RMContainerAllocator counts failed maps towards Reduce ramp up -- Key: MAPREDUCE-3490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3490 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3490.patch The RMContainerAllocator does not differentiate between failed and successful maps while calculating whether reduce tasks are ready to launch. Failed tasks are also counted towards total completed tasks. Example. 4 failed maps, 10 total maps. Map%complete = 4/14 * 100 instead of being 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3483) CapacityScheduler reserves container on same node as AM but can't ever use due to never enough avail memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3483: Assignee: Arun C Murthy CapacityScheduler reserves container on same node as AM but can't ever use due to never enough avail memory --- Key: MAPREDUCE-3483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3483 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker Saw a case where a job was stuck trying to get reducers. The issue is the capacity scheduler reserved a container on the same node as the application master but there wasn't ever enough memory to run the reducer on that node. Node total memory was 8G, Reducer needed 8G, AM was using 2G. This particular job had 10 reducers and it was stuck waiting on the one because the AM + reserved reducer memory was already over the queue limit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3524) Scan runtime is more than 1.5x slower in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3524: Assignee: Vinod Kumar Vavilapalli Scan runtime is more than 1.5x slower in 0.23 - Key: MAPREDUCE-3524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3524 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Karam Singh Assignee: Vinod Kumar Vavilapalli Priority: Blocker Scan runtime is more than 1.5X slower(almost 92% increased) in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3534) Compression benchmark run-time increased by 13% in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3534: Assignee: Vinod Kumar Vavilapalli Compression benchmark run-time increased by 13% in 0.23 --- Key: MAPREDUCE-3534 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3534 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Vinay Kumar Thota Assignee: Vinod Kumar Vavilapalli Priority: Blocker Compression runtime is increased by 13% as well as throughput is decreased by 24% in 0.23 when compared to 0.20.204 on a 350 node size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3525) Shuffle runtime is nearly 1.5x slower in 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3525: Assignee: Vinod Kumar Vavilapalli Shuffle runtime is nearly 1.5x slower in 0.23 - Key: MAPREDUCE-3525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3525 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Karam Singh Assignee: Vinod Kumar Vavilapalli Shuffle runtime is nearly 1.5X slower(almost 55% increased) in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3511) Counters occupy a good part of AM heap
[ https://issues.apache.org/jira/browse/MAPREDUCE-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3511: Assignee: Vinod Kumar Vavilapalli (was: Devaraj K) Devaraj - if you don't mind, I'll assign this to Vinod since he is blocked by this. Thanks. Counters occupy a good part of AM heap -- Key: MAPREDUCE-3511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3511 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Per task counters seem to be occupying a good part of an AMs heap. Looks like more than 50% of what's used by a TaskAttemptImpl object. This could be optimized by interning strings or possibly using mrv1 counters which are optimized. Currently counters are converted from mrv1 to mrv2 format for in memory storage. The conversion could be delayed till it's actually required for RPC transfers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3530) Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
[
https://issues.apache.org/jira/browse/MAPREDUCE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3530:
Assignee: Arun C Murthy (was: Vinod Kumar Vavilapalli)
I'll take a look.
Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
---
Key: MAPREDUCE-3530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3530
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager, scheduler
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Arun C Murthy
Priority: Blocker
Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
but ResourceManager keeps on running.
I have been observing intermitently for last 3 weeks.
But with latest svn code. I tried to run sort twice and both times Job got
stuck due to NPE.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerLaunchedOnNode(SchedulerApp.java:181)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.containerLaunchedOnNode(CapacityScheduler.java:596)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:539)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:617)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3399) ContainerLocalizer should request new resources after completing the current one
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3399: Assignee: Siddharth Seth ContainerLocalizer should request new resources after completing the current one Key: MAPREDUCE-3399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3530) Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
[
https://issues.apache.org/jira/browse/MAPREDUCE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3530:
Assignee: Vinod Kumar Vavilapalli
Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
---
Key: MAPREDUCE-3530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3530
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager, scheduler
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Sometimes NODE_UPDATE to the scheduler throws NPE causes scheduling to stop
but ResourceManager keeps on running.
I have been observing intermitently for last 3 weeks.
But with latest svn code. I tried to run sort twice and both times Job got
stuck due to NPE.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerLaunchedOnNode(SchedulerApp.java:181)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.containerLaunchedOnNode(CapacityScheduler.java:596)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:539)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:617)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-1118) Capacity Scheduler scheduling information is hard to read / should be tabular format
[ https://issues.apache.org/jira/browse/MAPREDUCE-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-1118: Assignee: Krishna Ramachandran (was: Milind Bhandarkar) Capacity Scheduler scheduling information is hard to read / should be tabular format Key: MAPREDUCE-1118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1118 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched Affects Versions: 0.20.2 Reporter: Allen Wittenauer Assignee: Krishna Ramachandran Fix For: 0.20.203.0, 0.22.0 Attachments: MR-1118-22.patch, mapred-1118-1.patch, mapred-1118-2.patch, mapred-1118-3.patch, mapred-1118.20S.patch, mapred-1118.patch The scheduling information provided by the capacity scheduler is extremely hard to read on the job tracker web page. Instead of just flat text, it should be presenting the information in a tabular format, similar to what the fair share scheduler provides. This makes it much easier to compare what different queues are doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3329) capacity schedule maximum-capacity allowed to be less then capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3329: Assignee: Arun C Murthy capacity schedule maximum-capacity allowed to be less then capacity --- Key: MAPREDUCE-3329 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3329 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker When configuring the capacity scheduler capacity and maximum-capacity, it allows the maximum-capacity to be less then the capacity. I did not test to see what true limit is, I assume maximum capacity. output from mapred queue -list where capacity = 10%, max capacity = 5%. Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3307) Improve logging on the console during job execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3307: Assignee: Arun C Murthy Improve logging on the console during job execution --- Key: MAPREDUCE-3307 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3307 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 There is a lot of redundant information being printed on the console and a not so intuitive flow of events. We should improve the logging on console during job execution. More details in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3265) Reduce log level on MR2 IPC construction, etc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3265: Assignee: Arun C Murthy Reduce log level on MR2 IPC construction, etc - Key: MAPREDUCE-3265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Arun C Murthy Priority: Blocker Currently MR's IPC logging is very verbose. For example, I see a lot of: 11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol 11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012 11/10/25 12:14:08 INFO mapred.ClientCache: Connecting to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 11/10/25 12:14:08 INFO mapred.ClientCache: Connected to HistoryServer at: c0309.hal.cloudera.com:10020 11/10/25 12:14:08 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol ... when submitting a job. This should be DEBUG level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3284) bin/mapred queue fails with JobQueueClient ClassNotFoundException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3284:
Assignee: Arun C Murthy
Trivial fix is to move JobQueueClient to hadoop-mapreduce-client-core. We
should have done this long ago...
bin/mapred queue fails with JobQueueClient ClassNotFoundException
-
Key: MAPREDUCE-3284
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3284
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Arun C Murthy
Fix For: 0.23.0
bin/mapred queue fails with the following exception:
{code}
-bash$ bin/mapred queue
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/JobQueueClient
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapred.JobQueueClient
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: org.apache.hadoop.mapred.JobQueueClient.
Program will exit.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3282) bin/mapred job -list throws exception
[ https://issues.apache.org/jira/browse/MAPREDUCE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3282: Assignee: Arun C Murthy bin/mapred job -list throws exception - Key: MAPREDUCE-3282 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3282 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Arun C Murthy Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-3282.patch bin/mapred job -list throws exception when mapreduce.framework.name is set to yarn -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3290) list-active-trackers throws NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3290: Assignee: Arun C Murthy Ramya, I can't seem to reproduce this with latest builds. Can you please let me know further details? Tx list-active-trackers throws NPE --- Key: MAPREDUCE-3290 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3290 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.0 bin/mapred -list-active-trackers throws NPE in mrV2. Trace in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2766) [MR-279] Set correct permissions for files in dist cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-2766: Assignee: Hitesh Shah [MR-279] Set correct permissions for files in dist cache Key: MAPREDUCE-2766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2766 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Currently, the files in both public and private dist cache are having 777 permission. Also, the group ownership of files on private cache have to be set to $TT_SPECIAL_GROUP -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-2821: Assignee: Mahadev konar (was: Harsh J) [MR-279] Missing fields in job summary logs Key: MAPREDUCE-2821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2821.patch The following fields are missing in the job summary logs in mrv2: - numSlotsPerMap - numSlotsPerReduce - clusterCapacity (Earlier known as clusterMapCapacity and clusterReduceCapacity in 0.20.x) The first two fields are important to know if the job was a High RAM job or not and the last field is important to know the total available resource in the cluster during job execution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3254) Streaming jobs failing with PipeMapRunner ClassNotFoundException
[ https://issues.apache.org/jira/browse/MAPREDUCE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3254: Assignee: Hitesh Shah Streaming jobs failing with PipeMapRunner ClassNotFoundException Key: MAPREDUCE-3254 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3254 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Hitesh Shah Priority: Blocker ClassNotFoundException: org.apache.hadoop.streaming.PipeMapRunner encountered while running streaming jobs. Stack trace in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3253) ContextFactory throw NoSuchFieldException
[ https://issues.apache.org/jira/browse/MAPREDUCE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3253: Assignee: Arun C Murthy (was: Mahadev konar) Ok, I can reproduce this on branch-0.23. ContextFactory throw NoSuchFieldException - Key: MAPREDUCE-3253 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3253 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Daniel Dai Assignee: Arun C Murthy Priority: Blocker I see exceptions from ContextFactory when I am running Pig unit test: Caused by: java.lang.IllegalArgumentException: Can't find field at org.apache.hadoop.mapreduce.ContextFactory.clinit(ContextFactory.java:139) Caused by: java.lang.NoSuchFieldException: reporter at java.lang.Class.getDeclaredField(Class.java:1882) at org.apache.hadoop.mapreduce.ContextFactory.clinit(ContextFactory.java:126) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2977) ResourceManager needs to renew and cancel tokens associated with a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-2977: Assignee: Arun C Murthy (was: Vinod Kumar Vavilapalli) ResourceManager needs to renew and cancel tokens associated with a job -- Key: MAPREDUCE-2977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2977 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Owen O'Malley Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 The JobTracker currently manages tokens for the applications and the resource manager needs the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3028: Assignee: Ravi Prakash (was: Arun C Murthy) Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3028.branch-0.23.patch, MAPREDUCE-3028.patch, MAPREDUCE-3028.patch, MAPREDUCE-3028.patch, MAPREDUCE-3028.patch, MAPREDUCE-3028.patch Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3143) Complete aggregation of user-logs spit out by containers onto DFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3143: Assignee: (was: Vinod Kumar Vavilapalli) Complete aggregation of user-logs spit out by containers onto DFS - Key: MAPREDUCE-3143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3143 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Fix For: 0.23.0 Already implemented the feature for handling user-logs spit out by containers in NodeManager. But the feature is currently disabled due to user-interface issues. This is the umbrella ticket for tracking the pending bugs w.r.t putting container-logs on DFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2746) [MR-279] [Security] Yarn servers can't communicate with each other with hadoop.security.authorization set to true
[
https://issues.apache.org/jira/browse/MAPREDUCE-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-2746:
Assignee: Arun C Murthy
[MR-279] [Security] Yarn servers can't communicate with each other with
hadoop.security.authorization set to true
-
Key: MAPREDUCE-2746
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2746
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, security
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Arun C Murthy
Priority: Blocker
Fix For: 0.23.0
Because of this problem, till now, we've been testing YARN+MR with
{{hadoop.security.authorization}} set to false. We need to register yarn
communication protocols in the implementation of the authorization related
PolicyProvider (MapReducePolicyProvider.java).
[~devaraj] also found this issue independently.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3178) Capacity Schedular shows incorrect cluster information in the RM logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3178: Assignee: Bhallamudi Venkata Siva Kamesh Bala, this is a good to fix bug. Can you pls add a unit test? Tx. Capacity Schedular shows incorrect cluster information in the RM logs - Key: MAPREDUCE-3178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3178 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-3178.patch When we start the NM, after stopping it (in a quick session) CS shows incorrect information about clusterResource in the logs. I have encountered this issue in a pseudo cluster mode and steps to reproduce are 1) start the YARN cluster 2) stop a NM and start the NM again (in a quick session) There should be a NM running in the cluster however as I observed RM detects NM as dead, after default time since its actual unavailability(In this case NM has been stopped). If you start your NM before this time (default time), ResourceTracker throws IOEx, however, CS adds the NM's capacity to the clusterResource. After elapsed time (default time) when RM detects NM as dead, RM removes the NM and hence capacity of the cluster will be subtracted by the amount NM capacity. Eventually there is no NM running in the cluster, but capacity of the cluster is NM's capacity (by default) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3034: Assignee: Arun C Murthy (was: Vinod Kumar Vavilapalli) NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3103) Implement Job ACLs for MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3103: Assignee: Vinod Kumar Vavilapalli Implement Job ACLs for MRAppMaster -- Key: MAPREDUCE-3103 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3103 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3058) Sometimes task keeps on running while its Syslog says that it is shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3058:
Assignee: Hitesh Shah (was: Vinod Kumar Vavilapalli)
Hitesh - can you please take a look since you are already working on related
areas?
Currently the TT sends SIGTERM followed by SIGKILL...
Sometimes task keeps on running while its Syslog says that it is shutdown
-
Key: MAPREDUCE-3058
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3058
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Affects Versions: 0.23.0
Reporter: Karam Singh
Assignee: Hitesh Shah
Priority: Critical
Fix For: 0.23.0
Attachments: MAPREDUCE-3058-20110923.txt
While running GridMixV3, one of the jobs got stuck for 15 hrs. After clicking
on the Job-page, found one of its reduces to be stuck. Looking at syslog of
the stuck reducer, found this:
Task-logs' head:
{code}
2011-09-19 17:57:22,002 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
at 10 second(s).
2011-09-19 17:57:22,002 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system
started
{code}
Task-logs' tail:
{code}
2011-09-19 18:06:49,818 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink as DATANODE1
2011-09-19 18:06:49,818 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
for block
BP-1405370709-NAMENODE-1316452621953:blk_-7004355226367468317_79871 in
pipeline DATANODE2, DATANODE1: bad datanode DATANODE1
2011-09-19 18:06:49,818 DEBUG
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtocol:
lastAckedSeqno = 26870
2011-09-19 18:06:49,820 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf sending #454
2011-09-19 18:06:49,826 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf got value #454
2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.ipc.RPC: Call:
getAdditionalDatanode 8
2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.hdfs.DFSClient: Connecting to
datanode DATANODE2
2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.hdfs.DFSClient: Send buf size
131071
2011-09-19 18:06:49,833 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:860)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:929)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:740)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:415)
2011-09-19 18:06:49,837 WARN org.apache.hadoop.mapred.YarnChild: Exception
running child : java.io.EOFException: Premature EOF: no length prefix
available
at
org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:860)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:929)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:740)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:415)
2011-09-19 18:06:49,837 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to APPMASTER from job_1316452677984_0862 sending #455
2011-09-19 18:06:49,839 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to APPMASTER from job_1316452677984_0862 got value
#455
2011-09-19 18:06:49,840 DEBUG org.apache.hadoop.ipc.RPC: Call: statusUpdate 3
2011-09-19 18:06:49,840 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
2011-09-19 18:06:49,840 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf sending #456
2011-09-19 18:06:49,858 DEBUG org.apache.hadoop.ipc.Client: IPC Client
(26613121) connection to NAMENODE from gridperf got value #456
2011-09-19 18:06:49,858 DEBUG
[jira] [Assigned] (MAPREDUCE-3126) mr job stuck because reducers using all slots and mapper isn't scheduled
[ https://issues.apache.org/jira/browse/MAPREDUCE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-3126: Assignee: Arun C Murthy I'll fix the CS corner case at least with this jira... mr job stuck because reducers using all slots and mapper isn't scheduled Key: MAPREDUCE-3126 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3126 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.0 The command in MAPREDUCE-3124 run and this job got hung with 1 Map task waiting for resources and 7 Reducers running (2 waiting). The mapper got scheduler, then AM scheduled the reducers, the map task failed and tried to start a new attempt but reducers were using all the slots. I will try to add some more info from the logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-2889: Assignee: Hitesh Shah Add docs for writing new application frameworks --- Key: MAPREDUCE-2889 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Priority: Critical Fix For: 0.23.0 We need to add docs for writing new application frameworks, including examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3084) race when KILL_CONTAINER is received for a LOCALIZED container
[
https://issues.apache.org/jira/browse/MAPREDUCE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned MAPREDUCE-3084:
Assignee: Hitesh Shah
race when KILL_CONTAINER is received for a LOCALIZED container
--
Key: MAPREDUCE-3084
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3084
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Hitesh Shah
Priority: Blocker
Depending on when ContainersLaunch starts a container, {{KILL_CONTAINER}}
when container state is {{LOCALIZED}} ({{LAUNCH_CONTAINER}} event already
sent) can end up generating a {{CONTAINER_LAUNCHED}} event - which isn't
handled by ContainerState: {{KILLING}}. Also, the launched container won't be
killed since {{CLEANUP_CONTAINER}} would have already been processed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2989) JobHistory should link to task logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-2989: Assignee: Siddharth Seth JobHistory should link to task logs --- Key: MAPREDUCE-2989 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2989 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical The log link on the task attempt page is currently broken - since it relies on a ContainerId. We should either pass the containerId via a history event - or some kind of field with information about the log location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2977) ResourceManager needs to renew and cancel tokens associated with a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-2977: Assignee: Vinod Kumar Vavilapalli ResourceManager needs to renew and cancel tokens associated with a job -- Key: MAPREDUCE-2977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2977 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Owen O'Malley Assignee: Vinod Kumar Vavilapalli Priority: Blocker The JobTracker currently manages tokens for the applications and the resource manager needs the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
