[jira] [Moved] (YARN-2862) RM might not start if the machine was hard shutdown and FileSystemRMStateStore was used
[
https://issues.apache.org/jira/browse/YARN-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma moved HADOOP-11305 to YARN-2862:
Key: YARN-2862 (was: HADOOP-11305)
Project: Hadoop YARN (was: Hadoop Common)
> RM might not start if the machine was hard shutdown and
> FileSystemRMStateStore was used
> ---
>
> Key: YARN-2862
> URL: https://issues.apache.org/jira/browse/YARN-2862
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ming Ma
>
> This might be a known issue. Given FileSystemRMStateStore isn't used for HA
> scenario, it might not be that important, unless there is something we need
> to fix at RM layer to make it more tolerant to RMStore issue.
> When RM was hard shutdown, OS might not get a chance to persist blocks. Some
> of the stored application data end up with size zero after reboot. And RM
> didn't like that.
> {noformat}
> ls -al
> /var/log/hadoop/rmstore/FSRMStateRoot/RMAppRoot/application_1412702189634_324351
> total 156
> drwxr-xr-x.2 x y 4096 Nov 13 16:45 .
> drwxr-xr-x. 1524 x y 151552 Nov 13 16:45 ..
> -rw-r--r--.1 x y 0 Nov 13 16:45
> appattempt_1412702189634_324351_01
> -rw-r--r--.1 x y 0 Nov 13 16:45
> .appattempt_1412702189634_324351_01.crc
> -rw-r--r--.1 x y 0 Nov 13 16:45 application_1412702189634_324351
> -rw-r--r--.1 x y 0 Nov 13 16:45 .application_1412702189634_324351.crc
> {noformat}
> When RM starts up
> {noformat}
> 2014-11-13 16:55:25,844 WARN org.apache.hadoop.fs.FSInputChecker: Problem
> opening checksum file:
> file:/var/log/hadoop/rmstore/FSRMStateRoot/RMAppRoot/application_1412702189634_324351/application_1412702189634_324351.
> Ignoring exception:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:792)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.readFile(FileSystemRMStateStore.java:501)
> ...
> 2014-11-13 17:40:48,876 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Failed to
> load/recover state
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ApplicationState.getAppId(RMStateStore.java:184)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:425)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1027)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:484)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:834)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2056) Disable preemption at Queue level
[
https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211663#comment-14211663
]
Hadoop QA commented on YARN-2056:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681445/YARN-2056.201411132215.txt
against trunk revision d005404.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5842//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5842//console
This message is automatically generated.
> Disable preemption at Queue level
> -
>
> Key: YARN-2056
> URL: https://issues.apache.org/jira/browse/YARN-2056
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Mayank Bansal
>Assignee: Eric Payne
> Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt,
> YARN-2056.201408310117.txt, YARN-2056.201409022208.txt,
> YARN-2056.201409181916.txt, YARN-2056.201409210049.txt,
> YARN-2056.201409232329.txt, YARN-2056.201409242210.txt,
> YARN-2056.201410132225.txt, YARN-2056.201410141330.txt,
> YARN-2056.201410232244.txt, YARN-2056.201410311746.txt,
> YARN-2056.201411041635.txt, YARN-2056.201411072153.txt,
> YARN-2056.201411122305.txt, YARN-2056.201411132215.txt
>
>
> We need to be able to disable preemption at individual queue level
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[
https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211664#comment-14211664
]
Hadoop QA commented on YARN-2556:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681427/yarn2556.patch
against trunk revision 1a2e5cb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.fs.slive.TestSlive
org.apache.hadoop.mapred.TestJobSysDirWithDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5840//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5840//console
This message is automatically generated.
> Tool to measure the performance of the timeline server
> --
>
> Key: YARN-2556
> URL: https://issues.apache.org/jira/browse/YARN-2556
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Jonathan Eagles
>Assignee: chang li
> Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch,
> yarn2556.patch, yarn2556.patch, yarn2556_wip.patch
>
>
> We need to be able to understand the capacity model for the timeline server
> to give users the tools they need to deploy a timeline server with the
> correct capacity.
> I propose we create a mapreduce job that can measure timeline server write
> and read performance. Transactions per second, I/O for both read and write
> would be a good start.
> This could be done as an example or test job that could be tied into gridmix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2664) Improve RM webapp to expose info about reservations.
[
https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211645#comment-14211645
]
Hadoop QA commented on YARN-2664:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681433/YARN-2664.2.patch
against trunk revision d005404.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 6
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebAppFairScheduler
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5841//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5841//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5841//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5841//console
This message is automatically generated.
> Improve RM webapp to expose info about reservations.
>
>
> Key: YARN-2664
> URL: https://issues.apache.org/jira/browse/YARN-2664
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Matteo Mazzucchelli
> Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch,
> YARN-2664.2.patch, YARN-2664.patch
>
>
> YARN-1051 provides a new functionality in the RM to ask for reservation on
> resources. Exposing this through the webapp GUI is important.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2056) Disable preemption at Queue level
[
https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211630#comment-14211630
]
Wangda Tan commented on YARN-2056:
--
Hi [~eepayne],
Thanks for updating the patch and explanation.
I think we should always round-up wQavail to make sure queue with lower
ideal_assigned percentage will be satisfied first. Instead of {{multiply}} you
should use {{multiplyAndNormalizeUp}} here. The factor should be resource=(1,1)
Since your patch already sort queues from least satisfied to most satisfied, so
doing a force round-up here is safe.
Actually I think we should consider minimum_allocation in preemption policy, we
can address that in a separated JIRA.
bq. The point of that test was to indicate that preemption levelization will
still happen even if the thing asking for resources is the one that is
untouchable. If you think this test is unnecessary, I will take it out.
Since the result is not changed before/after we set preemption queue, I think
it is unnecessary, I would suggest to take it out.
Thanks,
Wangda
> Disable preemption at Queue level
> -
>
> Key: YARN-2056
> URL: https://issues.apache.org/jira/browse/YARN-2056
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Mayank Bansal
>Assignee: Eric Payne
> Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt,
> YARN-2056.201408310117.txt, YARN-2056.201409022208.txt,
> YARN-2056.201409181916.txt, YARN-2056.201409210049.txt,
> YARN-2056.201409232329.txt, YARN-2056.201409242210.txt,
> YARN-2056.201410132225.txt, YARN-2056.201410141330.txt,
> YARN-2056.201410232244.txt, YARN-2056.201410311746.txt,
> YARN-2056.201411041635.txt, YARN-2056.201411072153.txt,
> YARN-2056.201411122305.txt, YARN-2056.201411132215.txt
>
>
> We need to be able to disable preemption at individual queue level
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211610#comment-14211610
]
Hadoop QA commented on YARN-2861:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681444/YARN-2861.1.patch
against trunk revision d005404.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5843//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5843//console
This message is automatically generated.
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-2861.1.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2857) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/YARN-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211574#comment-14211574
]
Mohammad Kamrul Islam commented on YARN-2857:
-
Can some (binding) people please review it?
> ConcurrentModificationException in ContainerLogAppender
> ---
>
> Key: YARN-2857
> URL: https://issues.apache.org/jira/browse/YARN-2857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
> MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch,
> YARN-2857.3.patch
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2056) Disable preemption at Queue level
[
https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-2056:
-
Attachment: YARN-2056.201411132215.txt
Thank you very much [~leftnoteasy]. Here is the patch with my changes.
{quote}
1) Typo of comment in cloneQueue:
2. The logic should be correct, but I think it might be simpler to say:
untouchableExtra = max(extra - childrenPreemptable, 0) and as same as the code.
{quote}
Comment changed
{quote}
3)
bq. public double getIdealPctOfGuaranteed(TempQueue q)
The method doesn't need to be public anymore
{quote}
Changed to {{private}}
{quote}
4) Does it possible there's only one queue in getMostUnderservedQueues?If so,
you should check if q2 is null
{quote}
Deep down in {{getIdealPctOfGuaranteed}}, it eventually does the null check,
but I added a null check in {{getMostUnderservedQueues}} as well. Better safe
than sorry :-)
{quote}
1) testDisablePreemptionOverCapPlusPending
Should disable queueB instead of queueA? Currently, the test will preempt from
appB not matter if preemption disabled for queueA or not
{quote}
The point of that test was to indicate that preemption levelization will still
happen even if the thing asking for resources is the one that is untouchable.
If you think this test is unnecessary, I will take it out.
{quote}
2) changes for t{{estHierarchicalLarge}}:
I'm a little concern about this change, even if we considering round error,
appA should be taken about 9-10 resources, 9->6 seems some potential bug caused
issue, could you double check if it works as expected? (Without affect the
normal preemption logic).
3) As above for {{testSkipAMContainer}}
I suggest you can take some investigation about why some original numbers need
to be changed, if it is just a round problem, that should be fine, but we
should avoid behavior changes.
{quote}
Yes, these changes are definitely due to rounding. There are 2 things that
cause the rounding problem:
# The new algorithm in {{computeFixpointAllocation}} is different, so you would
expect there to be differences in rounding.
#* One oddity about this algorithm is that it is still using
{{normalizedGuarantee}} to calculate {{wQavail}}.
#* if you have A and B, and B is less served than A, when it gets down to the
last one or two resources, it will try to multiply 1 or 2 by
{{normalizedGuarantee}} and will offer B 0, so the last 1 or 2 go to A.
#* This shows up noticeably in the unit tests where there are a smaller total
number of resources.
#* The algorithm could do a couple of things to clean this up.
#* it could always round the wQavail
#* It could also calculate how much resources it would take to get B up to the
level of the next thing on the underserved queue, and if there are unassigned
resources, just offer them to B.
# The other thing that is causing rounding is that when {{cloneQueues}} gets
{{root.getAbsoluteUsedCapacity()}}, it often comes out as something
0.524761581421 instead of 0.525, and so when that is used to calculate
{{current}}, it thinks a queue's current is really one less than what it should
be.
#* Something that could be done here is we could round to the nearest 3 decimal
places, for example. 3 decimal places is what shows up in the scheduler UI
metrics.
> Disable preemption at Queue level
> -
>
> Key: YARN-2056
> URL: https://issues.apache.org/jira/browse/YARN-2056
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Mayank Bansal
>Assignee: Eric Payne
> Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt,
> YARN-2056.201408310117.txt, YARN-2056.201409022208.txt,
> YARN-2056.201409181916.txt, YARN-2056.201409210049.txt,
> YARN-2056.201409232329.txt, YARN-2056.201409242210.txt,
> YARN-2056.201410132225.txt, YARN-2056.201410141330.txt,
> YARN-2056.201410232244.txt, YARN-2056.201410311746.txt,
> YARN-2056.201411041635.txt, YARN-2056.201411072153.txt,
> YARN-2056.201411122305.txt, YARN-2056.201411132215.txt
>
>
> We need to be able to disable preemption at individual queue level
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated YARN-2861:
--
Attachment: YARN-2861.1.patch
Straightforward change: creating separate set of configs for the timeline DT
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-2861.1.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2857) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/YARN-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211519#comment-14211519
]
Gera Shegalov commented on YARN-2857:
-
Clean Jenkins in the last build demonstrates that the patch fixes the
reproducer in the previous build:
{code}
testAppendInClose(org.apache.hadoop.yarn.TestContainerLogAppender) Time
elapsed: 0.066 sec <<< ERROR!
java.util.ConcurrentModificationException: null
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761)
at java.util.LinkedList$ListItr.next(LinkedList.java:696)
at
org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:81)
at
org.apache.hadoop.yarn.TestContainerLogAppender.testAppendInClose(TestContainerLogAppender.java:44)
{code}
. +1(non-binding)
> ConcurrentModificationException in ContainerLogAppender
> ---
>
> Key: YARN-2857
> URL: https://issues.apache.org/jira/browse/YARN-2857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
> MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch,
> YARN-2857.3.patch
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at ja
[jira] [Updated] (YARN-2664) Improve RM webapp to expose info about reservations.
[
https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Mazzucchelli updated YARN-2664:
--
Attachment: YARN-2664.2.patch
Hi Carlo.
I submitted a new patch. I followed your suggestions to improve the work.
{quote}
6) Did you tried any scale-up tests with hundreds of jobs or very jugged plans?
{quote}
I tried with one hundred jobs, more or less. I'm going to try with even more
jobs.
{quote}
7) Gathering from the scheduler the "actual' resources utilized by the jobs
running in the reservation, and visualizing them. Since you have some "past"
tracked in the plan (needed for the CapacityOverTimePolicy), adding tracking of
actual utilization, shouldn't be too hard, and could provide some useful
insight to users/operators.
{quote}
I have an idea how to make it. I'll let you know
> Improve RM webapp to expose info about reservations.
>
>
> Key: YARN-2664
> URL: https://issues.apache.org/jira/browse/YARN-2664
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Matteo Mazzucchelli
> Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch,
> YARN-2664.2.patch, YARN-2664.patch
>
>
> YARN-1051 provides a new functionality in the RM to ask for reservation on
> resources. Exposing this through the webapp GUI is important.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2856) Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTED
[
https://issues.apache.org/jira/browse/YARN-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211489#comment-14211489
]
Hudson commented on YARN-2856:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6543 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6543/])
YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state on
app recovery. Contributed by Rohith Sharmaks (jianhe: rev
d005404ef7211fe96ce1801ed267a249568540fd)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
> Application recovery throw InvalidStateTransitonException: Invalid event:
> ATTEMPT_KILLED at ACCEPTED
>
>
> Key: YARN-2856
> URL: https://issues.apache.org/jira/browse/YARN-2856
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Critical
> Fix For: 2.7.0
>
> Attachments: YARN-2856.1.patch, YARN-2856.patch
>
>
> It is observed that recovering an application with its attempt KILLED final
> state throw below exception. And application remain in accepted state forever.
> {code}
> 2014-11-12 02:34:10,602 | ERROR | AsyncDispatcher event handler | Can't
> handle this event at current state |
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:673)
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> ATTEMPT_KILLED at ACCEPTED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:90)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:730)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:714)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2857) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/YARN-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211457#comment-14211457
]
Hadoop QA commented on YARN-2857:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681381/YARN-2857.3.patch
against trunk revision 1a2e5cb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5839//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5839//console
This message is automatically generated.
> ConcurrentModificationException in ContainerLogAppender
> ---
>
> Key: YARN-2857
> URL: https://issues.apache.org/jira/browse/YARN-2857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
> MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch,
> YARN-2857.3.patch
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMet
[jira] [Updated] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated YARN-2556: --- Attachment: yarn2556.patch attempted modified patch according to build failure > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: chang li > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
Zhijie Shen created YARN-2861:
-
Summary: Timeline DT secret manager should not reuse the RM's
configs.
Key: YARN-2861
URL: https://issues.apache.org/jira/browse/YARN-2861
Project: Hadoop YARN
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
This is the configs for RM DT secret manager. We should create separate ones
for timeline DT only.
{code}
@Override
protected void serviceInit(Configuration conf) throws Exception {
long secretKeyInterval =
conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
long tokenMaxLifetime =
conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
long tokenRenewInterval =
conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
secretManager = new TimelineDelegationTokenSecretManager(secretKeyInterval,
tokenMaxLifetime, tokenRenewInterval,
360);
secretManager.startThreads();
serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
super.init(conf);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211358#comment-14211358 ] Carlo Curino commented on YARN-2009: I agree with your observation... the set of invariants/semantics (queue capacity, max-capacity, user quotas, apps priority, max-am-percentage, container size and multi-resources, etc..) cross product with preferences/optimizations (spare AMs, node labels, locality, minimize latency of jobs, etc..) makes for a vast space of possible policies... Considering the hierarchical nature of queues correctly in this makes for an even worse space. Beside the challenge of writing uber-policies that can handle all that, it is very hard to tune right. Even just the simplistic preemption we have today is confusing even very competent users (I know for a fact). I worry that more and more complexity will get to be unmanageable by most. In a sense I am growing fond of a notion of "explainability" of a system behavior, which favor systems that one can easily understand/predict the behavior of (to the cost of some optimality). To this purpose our cut-point in the early design of preemption was to say: "preemption should only kick in to correct large imbalances, and operate on a rather slow time-scale". The idea was to for example consider that if I am preempting 1k containers for you to get your capacity, locality would matter less... and so would many minor other issues like local priorities, locality, container sizes etc.. Overall, I think we should be use-case driven. If there is a clear "need" for complexity to cope with observed issues I think we can add it, but I would suggest we refrain from adding too many knobs based on hypothetical scenarios. If a need is not present yet, I would propose to require a "sizeable win" as a bar for adding knobs.. if we can demonstrate on some non-trivial experimental setup that a knob can deliver substantial value than maybe it's ok. > Priority support for preemption in ProportionalCapacityPreemptionPolicy > --- > > Key: YARN-2009 > URL: https://issues.apache.org/jira/browse/YARN-2009 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Devaraj K >Assignee: Sunil G > > While preempting containers based on the queue ideal assignment, we may need > to consider preempting the low priority application containers first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2766) ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers
[
https://issues.apache.org/jira/browse/YARN-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211359#comment-14211359
]
Hudson commented on YARN-2766:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6541 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6541/])
YARN-2766. Made ApplicationHistoryManager return a sorted list of apps,
attempts and containers. Contributed by Robert Kanter. (zjshen: rev
3648cb57c9f018a3a339c26f5a0ca2779485521a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
> ApplicationHistoryManager is expected to return a sorted list of
> apps/attempts/containers
> --
>
> Key: YARN-2766
> URL: https://issues.apache.org/jira/browse/YARN-2766
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.7.0
>
> Attachments: YARN-2766.patch, YARN-2766.patch, YARN-2766.patch,
> YARN-2766.patch
>
>
> {{TestApplicationHistoryClientService.testContainers}} and
> {{TestApplicationHistoryClientService.testApplicationAttempts}} both fail
> because the test assertions are assuming a returned Collection is in a
> certain order. The collection comes from a HashMap, so the order is not
> guaranteed, plus, according to [this
> page|http://docs.oracle.com/javase/8/docs/technotes/guides/collections/changes8.html],
> there are situations where the iteration order of a HashMap will be
> different between Java 7 and 8.
> We should fix the test code to not assume a specific ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2766) ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers
[
https://issues.apache.org/jira/browse/YARN-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211337#comment-14211337
]
Zhijie Shen commented on YARN-2766:
---
+1. Will commit the patch.
> ApplicationHistoryManager is expected to return a sorted list of
> apps/attempts/containers
> --
>
> Key: YARN-2766
> URL: https://issues.apache.org/jira/browse/YARN-2766
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-2766.patch, YARN-2766.patch, YARN-2766.patch,
> YARN-2766.patch
>
>
> {{TestApplicationHistoryClientService.testContainers}} and
> {{TestApplicationHistoryClientService.testApplicationAttempts}} both fail
> because the test assertions are assuming a returned Collection is in a
> certain order. The collection comes from a HashMap, so the order is not
> guaranteed, plus, according to [this
> page|http://docs.oracle.com/javase/8/docs/technotes/guides/collections/changes8.html],
> there are situations where the iteration order of a HashMap will be
> different between Java 7 and 8.
> We should fix the test code to not assume a specific ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211328#comment-14211328
]
Zhijie Shen commented on YARN-2859:
---
Binding the default port is not right for MiniYARNCluster. Will fix the problem.
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Priority: Critical
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen reassigned YARN-2859:
-
Assignee: Zhijie Shen
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Zhijie Shen
>Priority: Critical
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211323#comment-14211323 ] Hudson commented on YARN-2603: -- FAILURE: Integrated in Hadoop-trunk-Commit #6540 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6540/]) Revert "YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw)" (vinodkv: rev 4ae9780e6a05bfd6b93f1c871c22761ddd8b19cb) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/CHANGES.txt > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2603: -- Fix Version/s: (was: 2.7.0) Given enough time elapsed since my comment, I reverted this from trunk and branch-2. Will close this as Invalid in a day or two unless I hear otherwise. > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2860) MiniTezCluster needs to be cleaned up to remove MR dependencies
Hitesh Shah created YARN-2860: - Summary: MiniTezCluster needs to be cleaned up to remove MR dependencies Key: YARN-2860 URL: https://issues.apache.org/jira/browse/YARN-2860 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah There is a lot of old MR related code which is likely no longer need. For example, staging dir and job history utils. May be needed for testing MR compatibility but that could be handled explicitly only in cases where MR jobs are being run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
Hitesh Shah created YARN-2859: - Summary: ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster Key: YARN-2859 URL: https://issues.apache.org/jira/browse/YARN-2859 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Reporter: Hitesh Shah Priority: Critical In mini cluster, a random port should be used. Also, the config is not updated to the host that the process got bound to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated YARN-2859:
--
Description:
In mini cluster, a random port should be used.
Also, the config is not updated to the host that the process got bound to.
{code}
2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
(MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
address: localhost:10200
2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
(MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
web address: 0.0.0.0:8188
{code}
was:
In mini cluster, a random port should be used.
Also, the config is not updated to the host that the process got bound to.
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Priority: Critical
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (YARN-2838) Issues with TimeLineServer (Application History)
[ https://issues.apache.org/jira/browse/YARN-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210199#comment-14210199 ] Zhijie Shen edited comment on YARN-2838 at 11/13/14 7:34 PM: - bq. 1. Whatever the CLI command user executes is historyserver or timelineserver it looks like ApplicationHistoryServer only run. So can we modify the name of the class ApplicationHistoryServer to TimelineHistoryServer (or any other suitable name as it seems like any command user runs ApplicationHistoryServer is started) Yes, not just the the main entry point class, but the whole sub-module needs to be refactor somehow to reflect the generalized conception (YARN-2043). bq. 2. Instead of the "Starting the History Server anyway..." deprecated msg, can we have "Starting the Timeline History Server anyway...". bq. 3. Based on start or stop, deprecated message should get modified to "Starting the Timeline History Server anyway..." or "Stopping the Timeline History Server anyway..." See the comment before. bq. But any way we need to fix this issue also right ? so already any jira is raised or shall i work as part of this jira ? See YARN-2522. We can work this issue there. bq. And also please inform if this issue needs to be split into mulitple jiras (apart from documentation which you have already raised) would like to split and work on them. If you agree, we can close this Jira, and work on separate Jiras that focus on each individual issues. bq. As already i have started looking into these issues, was also planning to work on documentation. If you don't mind can you assign the issue (YARN-2854) to me ? No problem, assigned it to you. was (Author: zjshen): bq. 1. Whatever the CLI command user executes is historyserver or timelineserver it looks like ApplicationHistoryServer only run. So can we modify the name of the class ApplicationHistoryServer to TimelineHistoryServer (or any other suitable name as it seems like any command user runs ApplicationHistoryServer is started) Yes, not just the the main entry point class, but the whole sub-module needs to be refactor somehow to reflect the generalized conception (YARN-2043). bq. 2. Instead of the "Starting the History Server anyway..." deprecated msg, can we have "Starting the Timeline History Server anyway...". bq. 3. Based on start or stop, deprecated message should get modified to "Starting the Timeline History Server anyway..." or "Stopping the Timeline History Server anyway..." See the comment before. bq. But any way we need to fix this issue also right ? so already any jira is raised or shall i work as part of this jira ? See YARN-2522. We can work this issue there. bq. And also please inform if this issue needs to be split into mulitple jiras (apart from documentation which you have already raised) would like to split and work on them. If you agree, we can close this Jira, and work on separate Jiras that focus on each individual issues. bq. As already i have started looking into these issues, was also planning to work on documentation. If you don't mind can you assign the issue (YARN-2854) to me ? No problem, assigned it to you. > Issues with TimeLineServer (Application History) > > > Key: YARN-2838 > URL: https://issues.apache.org/jira/browse/YARN-2838 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: IssuesInTimelineServer.pdf > > > Few issues in usage of Timeline server for generic application history access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2838) Issues with TimeLineServer (Application History)
[ https://issues.apache.org/jira/browse/YARN-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2838: -- Affects Version/s: 2.6.0 > Issues with TimeLineServer (Application History) > > > Key: YARN-2838 > URL: https://issues.apache.org/jira/browse/YARN-2838 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: IssuesInTimelineServer.pdf > > > Few issues in usage of Timeline server for generic application history access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2838) Issues with TimeLineServer (Application History)
[ https://issues.apache.org/jira/browse/YARN-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210199#comment-14210199 ] Zhijie Shen commented on YARN-2838: --- bq. 1. Whatever the CLI command user executes is historyserver or timelineserver it looks like ApplicationHistoryServer only run. So can we modify the name of the class ApplicationHistoryServer to TimelineHistoryServer (or any other suitable name as it seems like any command user runs ApplicationHistoryServer is started) Yes, not just the the main entry point class, but the whole sub-module needs to be refactor somehow to reflect the generalized conception (YARN-2043). bq. 2. Instead of the "Starting the History Server anyway..." deprecated msg, can we have "Starting the Timeline History Server anyway...". bq. 3. Based on start or stop, deprecated message should get modified to "Starting the Timeline History Server anyway..." or "Stopping the Timeline History Server anyway..." See the comment before. bq. But any way we need to fix this issue also right ? so already any jira is raised or shall i work as part of this jira ? See YARN-2522. We can work this issue there. bq. And also please inform if this issue needs to be split into mulitple jiras (apart from documentation which you have already raised) would like to split and work on them. If you agree, we can close this Jira, and work on separate Jiras that focus on each individual issues. bq. As already i have started looking into these issues, was also planning to work on documentation. If you don't mind can you assign the issue (YARN-2854) to me ? No problem, assigned it to you. > Issues with TimeLineServer (Application History) > > > Key: YARN-2838 > URL: https://issues.apache.org/jira/browse/YARN-2838 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: IssuesInTimelineServer.pdf > > > Few issues in usage of Timeline server for generic application history access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2854) The document about timeline service and generic service needs to be updated
[ https://issues.apache.org/jira/browse/YARN-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2854: -- Assignee: Naganarasimha G R (was: Zhijie Shen) > The document about timeline service and generic service needs to be updated > --- > > Key: YARN-2854 > URL: https://issues.apache.org/jira/browse/YARN-2854 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2857) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/YARN-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Kamrul Islam updated YARN-2857:
Attachment: YARN-2857.3.patch
Added test case. Thanks [~jira.shegalov] for the help and comments.
> ConcurrentModificationException in ContainerLogAppender
> ---
>
> Key: YARN-2857
> URL: https://issues.apache.org/jira/browse/YARN-2857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
> MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch,
> YARN-2857.3.patch
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2858) TestRMHA#testFailoverAndTransitions fails in trunk against Java 8
Ted Yu created YARN-2858:
Summary: TestRMHA#testFailoverAndTransitions fails in trunk
against Java 8
Key: YARN-2858
URL: https://issues.apache.org/jira/browse/YARN-2858
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
>From https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/console :
{code}
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 51.034 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
testFailoverAndTransitions(org.apache.hadoop.yarn.server.resourcemanager.TestRMHA)
Time elapsed: 30.021 sec <<< ERROR!
java.lang.Exception: test timed out after 3 milliseconds
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:698)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:641)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1218)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at
com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.checkActiveRMWebServices(TestRMHA.java:157)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.checkActiveRMFunctionality(TestRMHA.java:142)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.testFailoverAndTransitions(TestRMHA.java:211)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2635) TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS
[ https://issues.apache.org/jira/browse/YARN-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210142#comment-14210142 ] Arun C Murthy commented on YARN-2635: - I've merged this back into branch-2.6 since it is safe, and is causing conflicts with too many cherry-picks. > TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS > -- > > Key: YARN-2635 > URL: https://issues.apache.org/jira/browse/YARN-2635 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wei Yan >Assignee: Wei Yan > Fix For: 2.6.0 > > Attachments: YARN-2635-1.patch, YARN-2635-2.patch, yarn-2635-3.patch, > yarn-2635-4.patch > > > If we change the scheduler from Capacity Scheduler to Fair Scheduler, the > TestRMRestart would fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2853) Killing app may hang while AM is unregistering
[ https://issues.apache.org/jira/browse/YARN-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210143#comment-14210143 ] Arun C Murthy commented on YARN-2853: - I've merged this back into branch-2.6 for hadoop-2.6.0-rc1. > Killing app may hang while AM is unregistering > -- > > Key: YARN-2853 > URL: https://issues.apache.org/jira/browse/YARN-2853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2853.1.patch, YARN-2853.1.patch, YARN-2853.2.patch, > YARN-2853.3.patch > > > When killing an app, app first moves to KILLING state, If RMAppAttempt > receives the attempt_unregister event before attempt_kill event, it'll > ignore the later attempt_kill event. Hence, RMApp won't be able to move to > KILLED state and stays at KILLING state forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2853) Killing app may hang while AM is unregistering
[ https://issues.apache.org/jira/browse/YARN-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-2853: Fix Version/s: (was: 2.7.0) 2.6.0 > Killing app may hang while AM is unregistering > -- > > Key: YARN-2853 > URL: https://issues.apache.org/jira/browse/YARN-2853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2853.1.patch, YARN-2853.1.patch, YARN-2853.2.patch, > YARN-2853.3.patch > > > When killing an app, app first moves to KILLING state, If RMAppAttempt > receives the attempt_unregister event before attempt_kill event, it'll > ignore the later attempt_kill event. Hence, RMApp won't be able to move to > KILLED state and stays at KILLING state forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2635) TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS
[ https://issues.apache.org/jira/browse/YARN-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-2635: Fix Version/s: (was: 2.7.0) 2.6.0 > TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS > -- > > Key: YARN-2635 > URL: https://issues.apache.org/jira/browse/YARN-2635 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wei Yan >Assignee: Wei Yan > Fix For: 2.6.0 > > Attachments: YARN-2635-1.patch, YARN-2635-2.patch, yarn-2635-3.patch, > yarn-2635-4.patch > > > If we change the scheduler from Capacity Scheduler to Fair Scheduler, the > TestRMRestart would fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2635) TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS
[ https://issues.apache.org/jira/browse/YARN-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210121#comment-14210121 ] Hudson commented on YARN-2635: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6536 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6536/]) YARN-2635. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 81dc0ac6dcf2f34ad607da815ea0144f178691a9) * hadoop-yarn-project/CHANGES.txt > TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS > -- > > Key: YARN-2635 > URL: https://issues.apache.org/jira/browse/YARN-2635 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wei Yan >Assignee: Wei Yan > Fix For: 2.7.0 > > Attachments: YARN-2635-1.patch, YARN-2635-2.patch, yarn-2635-3.patch, > yarn-2635-4.patch > > > If we change the scheduler from Capacity Scheduler to Fair Scheduler, the > TestRMRestart would fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2853) Killing app may hang while AM is unregistering
[ https://issues.apache.org/jira/browse/YARN-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210101#comment-14210101 ] Hudson commented on YARN-2853: -- FAILURE: Integrated in Hadoop-trunk-Commit #6535 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6535/]) YARN-2853. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev d648e60ebab7f1942dba92e9cd2cb62b8d70419b) * hadoop-yarn-project/CHANGES.txt > Killing app may hang while AM is unregistering > -- > > Key: YARN-2853 > URL: https://issues.apache.org/jira/browse/YARN-2853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.7.0 > > Attachments: YARN-2853.1.patch, YARN-2853.1.patch, YARN-2853.2.patch, > YARN-2853.3.patch > > > When killing an app, app first moves to KILLING state, If RMAppAttempt > receives the attempt_unregister event before attempt_kill event, it'll > ignore the later attempt_kill event. Hence, RMApp won't be able to move to > KILLED state and stays at KILLING state forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (YARN-2857) ConcurrentModificationException in ContainerLogAppender
[
https://issues.apache.org/jira/browse/YARN-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli moved MAPREDUCE-6139 to YARN-2857:
--
Target Version/s: 2.7.0 (was: 2.7.0)
Key: YARN-2857 (was: MAPREDUCE-6139)
Project: Hadoop YARN (was: Hadoop Map/Reduce)
> ConcurrentModificationException in ContainerLogAppender
> ---
>
> Key: YARN-2857
> URL: https://issues.apache.org/jira/browse/YARN-2857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch,
> MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2856) Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTED
[
https://issues.apache.org/jira/browse/YARN-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-2856:
--
Priority: Critical (was: Major)
Target Version/s: 2.7.0
> Application recovery throw InvalidStateTransitonException: Invalid event:
> ATTEMPT_KILLED at ACCEPTED
>
>
> Key: YARN-2856
> URL: https://issues.apache.org/jira/browse/YARN-2856
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Critical
> Attachments: YARN-2856.1.patch, YARN-2856.patch
>
>
> It is observed that recovering an application with its attempt KILLED final
> state throw below exception. And application remain in accepted state forever.
> {code}
> 2014-11-12 02:34:10,602 | ERROR | AsyncDispatcher event handler | Can't
> handle this event at current state |
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:673)
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> ATTEMPT_KILLED at ACCEPTED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:90)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:730)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:714)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2604) Scheduler should consider max-allocation-* in conjunction with the largest node
[ https://issues.apache.org/jira/browse/YARN-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209986#comment-14209986 ] Jason Lowe commented on YARN-2604: -- bq. Actually, I wonder if we should add a config to specify either (a) a particular number of NMs after which this behavior kicks in or (b) a minimum/floor value for the configurable maximum For the restart case this sounds a lot like YARN-2001 where we needed some kind of threshold to control when the RM started making scheduling decisions based on AM requests. This is a similar situation -- we don't want to make scheduling decisions too early before we have a good idea about the cluster. That uses a config, specific to work-preserving restart, to wait for 10s by default before acting on requests. We could do something similar here, either using the value directly or adding a "safe mode" config for the RM (and maybe tying one value to the other config by default since they are similar concepts). > Scheduler should consider max-allocation-* in conjunction with the largest > node > --- > > Key: YARN-2604 > URL: https://issues.apache.org/jira/browse/YARN-2604 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.5.1 >Reporter: Karthik Kambatla >Assignee: Robert Kanter > Attachments: YARN-2604.patch, YARN-2604.patch, YARN-2604.patch > > > If the scheduler max-allocation-* values are larger than the resources > available on the largest node in the cluster, an application requesting > resources between the two values will be accepted by the scheduler but the > requests will never be satisfied. The app essentially hangs forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2853) Killing app may hang while AM is unregistering
[ https://issues.apache.org/jira/browse/YARN-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209975#comment-14209975 ] Hudson commented on YARN-2853: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6534 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6534/]) YARN-2853. Fixed a bug in ResourceManager causing apps to hang when the user kill request races with ApplicationMaster finish. Contributed by Jian He. (vinodkv: rev 3651fe1b089851b38be351c00a9899817166bf3e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Killing app may hang while AM is unregistering > -- > > Key: YARN-2853 > URL: https://issues.apache.org/jira/browse/YARN-2853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.7.0 > > Attachments: YARN-2853.1.patch, YARN-2853.1.patch, YARN-2853.2.patch, > YARN-2853.3.patch > > > When killing an app, app first moves to KILLING state, If RMAppAttempt > receives the attempt_unregister event before attempt_kill event, it'll > ignore the later attempt_kill event. Hence, RMApp won't be able to move to > KILLED state and stays at KILLING state forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2846) Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart.
[
https://issues.apache.org/jira/browse/YARN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209974#comment-14209974
]
Hudson commented on YARN-2846:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #6534 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6534/])
YARN-2846. Incorrect persist exit code for running containers in
reacquireContainer() that interrupted by NodeManager restart. Contributed by
Junping Du (jlowe: rev 33ea5ae92b9dd3abace104903d9a94d17dd75af5)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/RecoveredContainerLaunch.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* hadoop-yarn-project/CHANGES.txt
> Incorrect persist exit code for running containers in reacquireContainer()
> that interrupted by NodeManager restart.
> ---
>
> Key: YARN-2846
> URL: https://issues.apache.org/jira/browse/YARN-2846
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: YARN-2846-demo.patch, YARN-2846.patch
>
>
> The NM restart work preserving feature could make running AM container get
> LOST and killed during stop NM daemon. The exception is like below:
> {code}
> 2014-11-11 00:48:35,214 INFO monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 22140 for
> container-id container_1415666714233_0001_01_84: 53.8 MB of 512 MB
> physical memory used; 931.3 MB of 1.0 GB virtual memory used
> 2014-11-11 00:48:35,223 ERROR nodemanager.NodeManager
> (SignalLogger.java:handle(60)) - RECEIVED SIGNAL 15: SIGTERM
> 2014-11-11 00:48:35,299 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
> HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50060
> 2014-11-11 00:48:35,337 INFO containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(512)) -
> Applications still running : [application_1415666714233_0001]
> 2014-11-11 00:48:35,338 INFO ipc.Server (Server.java:stop(2437)) - Stopping
> server on 45454
> 2014-11-11 00:48:35,344 INFO ipc.Server (Server.java:run(706)) - Stopping
> IPC Server listener on 45454
> 2014-11-11 00:48:35,346 INFO logaggregation.LogAggregationService
> (LogAggregationService.java:serviceStop(141)) -
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
> waiting for pending aggregation during exit
> 2014-11-11 00:48:35,347 INFO ipc.Server (Server.java:run(832)) - Stopping
> IPC Server Responder
> 2014-11-11 00:48:35,347 INFO logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:abortLogAggregation(502)) - Aborting log
> aggregation for application_1415666714233_0001
> 2014-11-11 00:48:35,348 WARN logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:run(382)) - Aggregation did not complete for
> application application_1415666714233_0001
> 2014-11-11 00:48:35,358 WARN monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:run(476)) -
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is interrupted. Exiting.
> 2014-11-11 00:48:35,406 ERROR launcher.RecoveredContainerLaunch
> (RecoveredContainerLaunch.java:call(87)) - Unable to recover container
> container_1415666714233_0001_01_01
> java.io.IOException: Interrupted while waiting for process 20001 to exit
> at
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:180)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:82)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:46)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.server.n
[jira] [Commented] (YARN-2846) Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart.
[
https://issues.apache.org/jira/browse/YARN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209948#comment-14209948
]
Jason Lowe commented on YARN-2846:
--
Agreed. Committing this.
> Incorrect persist exit code for running containers in reacquireContainer()
> that interrupted by NodeManager restart.
> ---
>
> Key: YARN-2846
> URL: https://issues.apache.org/jira/browse/YARN-2846
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Blocker
> Attachments: YARN-2846-demo.patch, YARN-2846.patch
>
>
> The NM restart work preserving feature could make running AM container get
> LOST and killed during stop NM daemon. The exception is like below:
> {code}
> 2014-11-11 00:48:35,214 INFO monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 22140 for
> container-id container_1415666714233_0001_01_84: 53.8 MB of 512 MB
> physical memory used; 931.3 MB of 1.0 GB virtual memory used
> 2014-11-11 00:48:35,223 ERROR nodemanager.NodeManager
> (SignalLogger.java:handle(60)) - RECEIVED SIGNAL 15: SIGTERM
> 2014-11-11 00:48:35,299 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
> HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50060
> 2014-11-11 00:48:35,337 INFO containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(512)) -
> Applications still running : [application_1415666714233_0001]
> 2014-11-11 00:48:35,338 INFO ipc.Server (Server.java:stop(2437)) - Stopping
> server on 45454
> 2014-11-11 00:48:35,344 INFO ipc.Server (Server.java:run(706)) - Stopping
> IPC Server listener on 45454
> 2014-11-11 00:48:35,346 INFO logaggregation.LogAggregationService
> (LogAggregationService.java:serviceStop(141)) -
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
> waiting for pending aggregation during exit
> 2014-11-11 00:48:35,347 INFO ipc.Server (Server.java:run(832)) - Stopping
> IPC Server Responder
> 2014-11-11 00:48:35,347 INFO logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:abortLogAggregation(502)) - Aborting log
> aggregation for application_1415666714233_0001
> 2014-11-11 00:48:35,348 WARN logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:run(382)) - Aggregation did not complete for
> application application_1415666714233_0001
> 2014-11-11 00:48:35,358 WARN monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:run(476)) -
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is interrupted. Exiting.
> 2014-11-11 00:48:35,406 ERROR launcher.RecoveredContainerLaunch
> (RecoveredContainerLaunch.java:call(87)) - Unable to recover container
> container_1415666714233_0001_01_01
> java.io.IOException: Interrupted while waiting for process 20001 to exit
> at
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:180)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:82)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:46)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:177)
> ... 6 more
> {code}
> In reacquireContainer() of ContainerExecutor.java, the while loop of checking
> container process (AM container) will be interrupted by NM stop. The
> IOException get thrown and failed to generate an ExitCodeFile for the running
> container. Later, the IOException will be caught in upper call
> (RecoveredContainerLaunch.call()) and the ExitCode (by default to be LOST
> without any setting) get persistent in NMStateStore.
> After NM restart again, this container is recovered as COMPLETE state but
> exit code is LOST (154) - cause this (AM) container get killed later.
> We should get rid of recording the exit code of running containers if
> detecting process is interrupted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened YARN-2603: --- This is not correct. We deliberately avoided putting compile time references to MapReduce in all of YARN. You should instead use yarn.nodemanager.env-whitelist and set HADOOP_MAPRED_HOME while starting nodemanager. OTOH, we are moving away from cluster installs of MapReduce to instead use DistributedCache: See MAPREDUCE-4421. I am going to wait a couple of hours before reverting this. > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2846) Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart.
[
https://issues.apache.org/jira/browse/YARN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209920#comment-14209920
]
Junping Du commented on YARN-2846:
--
bq. If we're going to kill normal containers on shutdown then why wouldn't we
also kill containers we are recovering as well? For the NM restart scenario
we're not supposed to be killing any containers.
My bad. Sorry for my expression above which isn't right. Yes. We shouldn't kill
containers for normal containers (fresh) and recovered containers (survival
from NM restart before).
bq. it's essentially a question of why doesn't interrupting the
ContainerLaunch thread manifest as a container completing as it did for a
recovered container.
Agree this is important. For ContainerLaunch (take DefaultContainerExecutor as
an example), I think thread are blocking in launchContainer()
{code}
if (isContainerActive(containerId)) {
shExec.execute();
}
{code}
The shExec.execute() will call Shell.runCommand() with building a new process
for the command (with an error monitoring thread). The thread will be waiting
at :
{code}
// wait for the process to finish and check the exit code
exitCode = process.waitFor();
{code}
It is also possible for InterruptedException get thrown there but the trigger
event is not the kill of NM but kill of shell process (so not affected by NM
kill). That may be the root cause for the different behavior now for fresh
container and recovered container. This is not my final conclusion, but I
would prefer to fix the existing significant bug (block container recovery for
recovered containers) here and we can do more investigation later. [~jlowe],
what do you think?
> Incorrect persist exit code for running containers in reacquireContainer()
> that interrupted by NodeManager restart.
> ---
>
> Key: YARN-2846
> URL: https://issues.apache.org/jira/browse/YARN-2846
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Blocker
> Attachments: YARN-2846-demo.patch, YARN-2846.patch
>
>
> The NM restart work preserving feature could make running AM container get
> LOST and killed during stop NM daemon. The exception is like below:
> {code}
> 2014-11-11 00:48:35,214 INFO monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 22140 for
> container-id container_1415666714233_0001_01_84: 53.8 MB of 512 MB
> physical memory used; 931.3 MB of 1.0 GB virtual memory used
> 2014-11-11 00:48:35,223 ERROR nodemanager.NodeManager
> (SignalLogger.java:handle(60)) - RECEIVED SIGNAL 15: SIGTERM
> 2014-11-11 00:48:35,299 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
> HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50060
> 2014-11-11 00:48:35,337 INFO containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(512)) -
> Applications still running : [application_1415666714233_0001]
> 2014-11-11 00:48:35,338 INFO ipc.Server (Server.java:stop(2437)) - Stopping
> server on 45454
> 2014-11-11 00:48:35,344 INFO ipc.Server (Server.java:run(706)) - Stopping
> IPC Server listener on 45454
> 2014-11-11 00:48:35,346 INFO logaggregation.LogAggregationService
> (LogAggregationService.java:serviceStop(141)) -
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
> waiting for pending aggregation during exit
> 2014-11-11 00:48:35,347 INFO ipc.Server (Server.java:run(832)) - Stopping
> IPC Server Responder
> 2014-11-11 00:48:35,347 INFO logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:abortLogAggregation(502)) - Aborting log
> aggregation for application_1415666714233_0001
> 2014-11-11 00:48:35,348 WARN logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:run(382)) - Aggregation did not complete for
> application application_1415666714233_0001
> 2014-11-11 00:48:35,358 WARN monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:run(476)) -
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is interrupted. Exiting.
> 2014-11-11 00:48:35,406 ERROR launcher.RecoveredContainerLaunch
> (RecoveredContainerLaunch.java:call(87)) - Unable to recover container
> container_1415666714233_0001_01_01
> java.io.IOException: Interrupted while waiting for process 20001 to exit
> at
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:180)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLau
[jira] [Commented] (YARN-2794) Fix log msgs about distributing system-credentials
[ https://issues.apache.org/jira/browse/YARN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209895#comment-14209895 ] Hudson commented on YARN-2794: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. (zjshen: rev be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Fix log msgs about distributing system-credentials > --- > > Key: YARN-2794 > URL: https://issues.apache.org/jira/browse/YARN-2794 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2794.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209908#comment-14209908 ] Hudson commented on YARN-2841: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 3e7a5157f05ebc83a57b799518dd2006f204d7a1) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2236) Shared Cache uploader service on the Node Manager
[ https://issues.apache.org/jira/browse/YARN-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209902#comment-14209902 ] Hudson commented on YARN-2236: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java > Shared Cache uploader service on the Node Manager > - > > Key: YARN-2236 > URL: https://issues.apache.org/jira/browse/YARN-2236 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2236-trunk-v1.patch, YARN-2236-trunk-v2.patch, > YARN-2236-trunk-v3.patch, YARN-2236-trunk-v4.patch, YARN-2236-trunk-v5.patch, > YARN-2236-trunk-v6.patch > > > Implement the shared cache uploader service on the node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209894#comment-14209894 ] Hudson commented on YARN-1492: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large clust
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209905#comment-14209905 ] Hudson commented on YARN-2843: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 51957ca81886ac3553c44848631fdf53766a25ae) * hadoop-yarn-project/CHANGES.txt > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Fix For: 2.6.0 > > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209900#comment-14209900 ] Hudson commented on YARN-2780: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne (jlowe: rev f8aefa5e9c8c6d2817205b5ed8d914db31f56ae7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * hadoop-yarn-project/CHANGES.txt > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-2780.v1.201411031728.txt, > YARN-2780.v2.201411061601.txt > > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209901#comment-14209901 ] Hudson commented on YARN-2603: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/4/]) YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) (aw: rev b0a9cd37ab9ef45ed9a71b04f1634fa54dd4c438) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209885#comment-14209885 ] Hudson commented on YARN-2603: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) (aw: rev b0a9cd37ab9ef45ed9a71b04f1634fa54dd4c438) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/CHANGES.txt > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209878#comment-14209878 ] Hudson commented on YARN-1492: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, so
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209892#comment-14209892 ] Hudson commented on YARN-2841: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 3e7a5157f05ebc83a57b799518dd2006f204d7a1) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2236) Shared Cache uploader service on the Node Manager
[ https://issues.apache.org/jira/browse/YARN-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209886#comment-14209886 ] Hudson commented on YARN-2236: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java > Shared Cache uploader service on the Node Manager > - > > Key: YARN-2236 > URL: https://issues.apache.org/jira/browse/YARN-2236 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2236-trunk-v1.patch, YARN-2236-trunk-v2.patch, > YARN-2236-trunk-v3.patch, YARN-2236-trunk-v4.patch, YARN-2236-trunk-v5.patch, > YARN-2236-trunk-v6.patch > > > Implement the shared cache uploader service on the node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209884#comment-14209884 ] Hudson commented on YARN-2780: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne (jlowe: rev f8aefa5e9c8c6d2817205b5ed8d914db31f56ae7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-2780.v1.201411031728.txt, > YARN-2780.v2.201411061601.txt > > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2794) Fix log msgs about distributing system-credentials
[ https://issues.apache.org/jira/browse/YARN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209879#comment-14209879 ] Hudson commented on YARN-2794: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. (zjshen: rev be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Fix log msgs about distributing system-credentials > --- > > Key: YARN-2794 > URL: https://issues.apache.org/jira/browse/YARN-2794 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2794.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209889#comment-14209889 ] Hudson commented on YARN-2843: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1956/]) YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 51957ca81886ac3553c44848631fdf53766a25ae) * hadoop-yarn-project/CHANGES.txt > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Fix For: 2.6.0 > > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209802#comment-14209802 ] Hudson commented on YARN-2841: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 3e7a5157f05ebc83a57b799518dd2006f204d7a1) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209795#comment-14209795 ] Hudson commented on YARN-2843: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 51957ca81886ac3553c44848631fdf53766a25ae) * hadoop-yarn-project/CHANGES.txt > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Fix For: 2.6.0 > > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209801#comment-14209801 ] Hudson commented on YARN-2841: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 3e7a5157f05ebc83a57b799518dd2006f204d7a1) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209796#comment-14209796 ] Hudson commented on YARN-2843: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 51957ca81886ac3553c44848631fdf53766a25ae) * hadoop-yarn-project/CHANGES.txt > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Fix For: 2.6.0 > > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2794) Fix log msgs about distributing system-credentials
[ https://issues.apache.org/jira/browse/YARN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209775#comment-14209775 ] Hudson commented on YARN-2794: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. (zjshen: rev be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java > Fix log msgs about distributing system-credentials > --- > > Key: YARN-2794 > URL: https://issues.apache.org/jira/browse/YARN-2794 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2794.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2236) Shared Cache uploader service on the Node Manager
[ https://issues.apache.org/jira/browse/YARN-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209790#comment-14209790 ] Hudson commented on YARN-2236: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java > Shared Cache uploader service on the Node Manager > - > > Key: YARN-2236 > URL: https://issues.apache.org/jira/browse/YARN-2236 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2236-trunk-v1.patch, YARN-2236-trunk-v2.patch, > YARN-2236-trunk-v3.patch, YARN-2236-trunk-v4.patch, YARN-2236-trunk-v5.patch, > YARN-2236-trunk-v6.patch > > > Implement the shared cache uploader service on the node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209774#comment-14209774 ] Hudson commented on YARN-1492: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes co
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209787#comment-14209787 ] Hudson commented on YARN-2603: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) (aw: rev b0a9cd37ab9ef45ed9a71b04f1634fa54dd4c438) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * hadoop-yarn-project/CHANGES.txt > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2794) Fix log msgs about distributing system-credentials
[ https://issues.apache.org/jira/browse/YARN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209776#comment-14209776 ] Hudson commented on YARN-2794: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. (zjshen: rev be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Fix log msgs about distributing system-credentials > --- > > Key: YARN-2794 > URL: https://issues.apache.org/jira/browse/YARN-2794 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2794.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2236) Shared Cache uploader service on the Node Manager
[ https://issues.apache.org/jira/browse/YARN-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209789#comment-14209789 ] Hudson commented on YARN-2236: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java > Shared Cache uploader service on the Node Manager > - > > Key: YARN-2236 > URL: https://issues.apache.org/jira/browse/YARN-2236 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2236-trunk-v1.patch, YARN-2236-trunk-v2.patch, > YARN-2236-trunk-v3.patch, YARN-2236-trunk-v4.patch, YARN-2236-trunk-v5.patch, > YARN-2236-trunk-v6.patch > > > Implement the shared cache uploader service on the node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209785#comment-14209785 ] Hudson commented on YARN-2780: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1932/]) YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne (jlowe: rev f8aefa5e9c8c6d2817205b5ed8d914db31f56ae7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-2780.v1.201411031728.txt, > YARN-2780.v2.201411061601.txt > > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209788#comment-14209788 ] Hudson commented on YARN-2603: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) (aw: rev b0a9cd37ab9ef45ed9a71b04f1634fa54dd4c438) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209786#comment-14209786 ] Hudson commented on YARN-2780: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne (jlowe: rev f8aefa5e9c8c6d2817205b5ed8d914db31f56ae7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-2780.v1.201411031728.txt, > YARN-2780.v2.201411061601.txt > > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209773#comment-14209773 ] Hudson commented on YARN-1492: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/4/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, someti
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209651#comment-14209651 ] Hudson commented on YARN-2841: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 3e7a5157f05ebc83a57b799518dd2006f204d7a1) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209636#comment-14209636 ] Hudson commented on YARN-1492: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes copy
[jira] [Commented] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209642#comment-14209642 ] Hudson commented on YARN-2780: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne (jlowe: rev f8aefa5e9c8c6d2817205b5ed8d914db31f56ae7) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-2780.v1.201411031728.txt, > YARN-2780.v2.201411061601.txt > > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2236) Shared Cache uploader service on the Node Manager
[ https://issues.apache.org/jira/browse/YARN-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209644#comment-14209644 ] Hudson commented on YARN-2236: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java > Shared Cache uploader service on the Node Manager > - > > Key: YARN-2236 > URL: https://issues.apache.org/jira/browse/YARN-2236 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2236-trunk-v1.patch, YARN-2236-trunk-v2.patch, > YARN-2236-trunk-v3.patch, YARN-2236-trunk-v4.patch, YARN-2236-trunk-v5.patch, > YARN-2236-trunk-v6.patch > > > Implement the shared cache uploader service on the node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2794) Fix log msgs about distributing system-credentials
[ https://issues.apache.org/jira/browse/YARN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209637#comment-14209637 ] Hudson commented on YARN-2794: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. (zjshen: rev be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/CHANGES.txt > Fix log msgs about distributing system-credentials > --- > > Key: YARN-2794 > URL: https://issues.apache.org/jira/browse/YARN-2794 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2794.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209643#comment-14209643 ] Hudson commented on YARN-2603: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) (aw: rev b0a9cd37ab9ef45ed9a71b04f1634fa54dd4c438) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209647#comment-14209647 ] Hudson commented on YARN-2843: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #742 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/742/]) YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 51957ca81886ac3553c44848631fdf53766a25ae) * hadoop-yarn-project/CHANGES.txt > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Fix For: 2.6.0 > > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2780) Log aggregated resource allocation in rm-appsummary.log
[ https://issues.apache.org/jira/browse/YARN-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209612#comment-14209612 ] Hudson commented on YARN-2780: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne (jlowe: rev f8aefa5e9c8c6d2817205b5ed8d914db31f56ae7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * hadoop-yarn-project/CHANGES.txt > Log aggregated resource allocation in rm-appsummary.log > --- > > Key: YARN-2780 > URL: https://issues.apache.org/jira/browse/YARN-2780 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Koji Noguchi >Assignee: Eric Payne >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-2780.v1.201411031728.txt, > YARN-2780.v2.201411061601.txt > > > YARN-415 added useful information about resource usage by applications. > Asking to log that info inside rm-appsummary.log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209617#comment-14209617 ] Hudson commented on YARN-2843: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 51957ca81886ac3553c44848631fdf53766a25ae) * hadoop-yarn-project/CHANGES.txt > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Fix For: 2.6.0 > > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209606#comment-14209606 ] Hudson commented on YARN-1492: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java > truly shared cache for jars (jobjar/libjar) > --- > > Key: YARN-1492 > URL: https://issues.apache.org/jira/browse/YARN-1492 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Chris Trezzo >Priority: Critical > Attachments: YARN-1492-all-trunk-v1.patch, > YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch, > YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, > shared_cache_design.pdf, shared_cache_design_v2.pdf, > shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, > shared_cache_design_v5.pdf, shared_cache_design_v6.pdf > > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, someti
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209620#comment-14209620 ] Hudson commented on YARN-2841: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. (acmurthy: rev 3e7a5157f05ebc83a57b799518dd2006f204d7a1) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2603) ApplicationConstants missing HADOOP_MAPRED_HOME
[ https://issues.apache.org/jira/browse/YARN-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209613#comment-14209613 ] Hudson commented on YARN-2603: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) (aw: rev b0a9cd37ab9ef45ed9a71b04f1634fa54dd4c438) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > ApplicationConstants missing HADOOP_MAPRED_HOME > --- > > Key: YARN-2603 > URL: https://issues.apache.org/jira/browse/YARN-2603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Allen Wittenauer >Assignee: Ray Chiang > Labels: newbie > Fix For: 2.7.0 > > Attachments: YARN-2603-01.patch > > > The Environment enum should have HADOOP_MAPRED_HOME listed as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2236) Shared Cache uploader service on the Node Manager
[ https://issues.apache.org/jira/browse/YARN-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209614#comment-14209614 ] Hudson commented on YARN-2236: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) (kasha: rev a04143039e7fe310d807f40584633096181cfada) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/ChecksumSHA256Impl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/TestSharedCacheUploader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksum.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourceRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LocalResourcePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/sharedcache/SharedCacheChecksumFactory.java > Shared Cache uploader service on the Node Manager > - > > Key: YARN-2236 > URL: https://issues.apache.org/jira/browse/YARN-2236 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2236-trunk-v1.patch, YARN-2236-trunk-v2.patch, > YARN-2236-trunk-v3.patch, YARN-2236-trunk-v4.patch, YARN-2236-trunk-v5.patch, > YARN-2236-trunk-v6.patch > > > Implement the shared cache uploader service on the node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2794) Fix log msgs about distributing system-credentials
[ https://issues.apache.org/jira/browse/YARN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209607#comment-14209607 ] Hudson commented on YARN-2794: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #4 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/4/]) YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. (zjshen: rev be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/CHANGES.txt > Fix log msgs about distributing system-credentials > --- > > Key: YARN-2794 > URL: https://issues.apache.org/jira/browse/YARN-2794 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.6.0 > > Attachments: YARN-2794.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2802) add AM container launch and register delay metrics in QueueMetrics to help diagnose performance issue.
[
https://issues.apache.org/jira/browse/YARN-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209474#comment-14209474
]
Hadoop QA commented on YARN-2802:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12681272/YARN-2802.002.patch
against trunk revision 7dae5b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5838//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5838//console
This message is automatically generated.
> add AM container launch and register delay metrics in QueueMetrics to help
> diagnose performance issue.
> --
>
> Key: YARN-2802
> URL: https://issues.apache.org/jira/browse/YARN-2802
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.5.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: YARN-2802.000.patch, YARN-2802.001.patch,
> YARN-2802.002.patch
>
>
> add AM container launch and register delay metrics in QueueMetrics to help
> diagnose performance issue.
> Added two metrics in QueueMetrics:
> aMLaunchDelay: the time spent from sending event AMLauncherEventType.LAUNCH
> to receiving event RMAppAttemptEventType.LAUNCHED in RMAppAttemptImpl.
> aMRegisterDelay: the time waiting from receiving event
> RMAppAttemptEventType.LAUNCHED to receiving event
> RMAppAttemptEventType.REGISTERED(ApplicationMasterService#registerApplicationMaster)
> in RMAppAttemptImpl.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
