[jira] [Commented] (YARN-90) NodeManager should identify failed disks becoming good back again
[ https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172050#comment-14172050 ] Ming Ma commented on YARN-90: - Thanks Varun. You and Jason discussed about disk clean up scenario. It will be useful to clarify if the following scenario will be resolved by this jira or a separate jira is necessary. 1. A disk became ready only. So DiskChecker will mark it as DiskErrorCause.OTHER. 2. Later the disk was repaired and became good. There are still data left on the disk. 3. Given these data are from old containers which have finished, who will clean up these data? Nit: disksTurnedBad's parameter name preCheckDirs, it is better to name it preFailedDirs. In the getDisksHealthReport, people can't tell if the disk fails due to full disk or failed disk, might be useful to distinguish the two cases. verifyDirUsingMkdir, is it necessary given DiskChecker.checkDir will check it? NodeManager should identify failed disks becoming good back again - Key: YARN-90 URL: https://issues.apache.org/jira/browse/YARN-90 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Ravi Gummadi Assignee: Varun Vasudev Attachments: YARN-90.1.patch, YARN-90.patch, YARN-90.patch, YARN-90.patch, YARN-90.patch, apache-yarn-90.0.patch, apache-yarn-90.1.patch, apache-yarn-90.2.patch, apache-yarn-90.3.patch, apache-yarn-90.4.patch, apache-yarn-90.5.patch, apache-yarn-90.6.patch, apache-yarn-90.7.patch, apache-yarn-90.8.patch, apache-yarn-90.9.patch MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes down, it is marked as failed forever. To reuse that disk (after it becomes good), NodeManager needs restart. This JIRA is to improve NodeManager to reuse good disks(which could be bad some time back). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2673) Add retry for timeline client
[
https://issues.apache.org/jira/browse/YARN-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172052#comment-14172052
]
Hadoop QA commented on YARN-2673:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674925/YARN-2673-101414-2.patch
against trunk revision 1220bb7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5401//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5401//console
This message is automatically generated.
Add retry for timeline client
-
Key: YARN-2673
URL: https://issues.apache.org/jira/browse/YARN-2673
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Attachments: YARN-2673-101414-1.patch, YARN-2673-101414-2.patch,
YARN-2673-101414.patch
Timeline client now does not handle the case gracefully when the server is
down. Jobs from distributed shell may fail due to ATS restart. We may need to
add some retry mechanisms to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: (was: YARN-2198.12.patch) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor -- Key: YARN-2198 URL: https://issues.apache.org/jira/browse/YARN-2198 Project: Hadoop YARN Issue Type: Improvement Reporter: Remus Rusanu Assignee: Remus Rusanu Labels: security, windows Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, YARN-2198.11.patch, YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch YARN-1972 introduces a Secure Windows Container Executor. However this executor requires the process launching the container to be LocalSystem or a member of the a local Administrators group. Since the process in question is the NodeManager, the requirement translates to the entire NM to run as a privileged account, a very large surface area to review and protect. This proposal is to move the privileged operations into a dedicated NT service. The NM can run as a low privilege account and communicate with the privileged NT service when it needs to launch a container. This would reduce the surface exposed to the high privileges. There has to exist a secure, authenticated and authorized channel of communication between the NM and the privileged NT service. Possible alternatives are a new TCP endpoint, Java RPC etc. My proposal though would be to use Windows LPC (Local Procedure Calls), which is a Windows platform specific inter-process communication channel that satisfies all requirements and is easy to deploy. The privileged NT service would register and listen on an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop with libwinutils which would host the LPC client code. The client would connect to the LPC port (NtConnectPort) and send a message requesting a container launch (NtRequestWaitReplyPort). LPC provides authentication and the privileged NT service can use authorization API (AuthZ) to validate the caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.12.patch Remove the need to run NodeManager as privileged account for Windows Secure Container Executor -- Key: YARN-2198 URL: https://issues.apache.org/jira/browse/YARN-2198 Project: Hadoop YARN Issue Type: Improvement Reporter: Remus Rusanu Assignee: Remus Rusanu Labels: security, windows Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch YARN-1972 introduces a Secure Windows Container Executor. However this executor requires the process launching the container to be LocalSystem or a member of the a local Administrators group. Since the process in question is the NodeManager, the requirement translates to the entire NM to run as a privileged account, a very large surface area to review and protect. This proposal is to move the privileged operations into a dedicated NT service. The NM can run as a low privilege account and communicate with the privileged NT service when it needs to launch a container. This would reduce the surface exposed to the high privileges. There has to exist a secure, authenticated and authorized channel of communication between the NM and the privileged NT service. Possible alternatives are a new TCP endpoint, Java RPC etc. My proposal though would be to use Windows LPC (Local Procedure Calls), which is a Windows platform specific inter-process communication channel that satisfies all requirements and is easy to deploy. The privileged NT service would register and listen on an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop with libwinutils which would host the LPC client code. The client would connect to the LPC port (NtConnectPort) and send a message requesting a container launch (NtRequestWaitReplyPort). LPC provides authentication and the privileged NT service can use authorization API (AuthZ) to validate the caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[
https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172064#comment-14172064
]
Hadoop QA commented on YARN-2198:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674933/YARN-2198.12.patch
against trunk revision 1220bb7.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5402//console
This message is automatically generated.
Remove the need to run NodeManager as privileged account for Windows Secure
Container Executor
--
Key: YARN-2198
URL: https://issues.apache.org/jira/browse/YARN-2198
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Remus Rusanu
Assignee: Remus Rusanu
Labels: security, windows
Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch,
YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.2.patch, YARN-2198.3.patch,
YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch,
YARN-2198.delta.7.patch, YARN-2198.separation.patch,
YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch,
YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch
YARN-1972 introduces a Secure Windows Container Executor. However this
executor requires the process launching the container to be LocalSystem or a
member of the a local Administrators group. Since the process in question is
the NodeManager, the requirement translates to the entire NM to run as a
privileged account, a very large surface area to review and protect.
This proposal is to move the privileged operations into a dedicated NT
service. The NM can run as a low privilege account and communicate with the
privileged NT service when it needs to launch a container. This would reduce
the surface exposed to the high privileges.
There has to exist a secure, authenticated and authorized channel of
communication between the NM and the privileged NT service. Possible
alternatives are a new TCP endpoint, Java RPC etc. My proposal though would
be to use Windows LPC (Local Procedure Calls), which is a Windows platform
specific inter-process communication channel that satisfies all requirements
and is easy to deploy. The privileged NT service would register and listen on
an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop
with libwinutils which would host the LPC client code. The client would
connect to the LPC port (NtConnectPort) and send a message requesting a
container launch (NtRequestWaitReplyPort). LPC provides authentication and
the privileged NT service can use authorization API (AuthZ) to validate the
caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-570) Time strings are formated in different timezone
[ https://issues.apache.org/jira/browse/YARN-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172065#comment-14172065 ] Akira AJISAKA commented on YARN-570: Thanks [~rchiang] for the comment. Fixed the bug by padding zero. bq. it could be confusing in a completely different way. It's a good idea to show the local time of the browser everywhere, however, there needs some code change (rendering date by JavaScript instead of Java) to achieve this. Hopefully, I'll post a new patch in the end of the week. Time strings are formated in different timezone --- Key: YARN-570 URL: https://issues.apache.org/jira/browse/YARN-570 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.2.0 Reporter: Peng Zhang Assignee: Akira AJISAKA Attachments: MAPREDUCE-5141.patch, YARN-570.2.patch, YARN-570.3.patch, YARN-570.4.patch, YARN-570.5.patch Time strings on different page are displayed in different timezone. If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as Wed, 10 Apr 2013 08:29:56 GMT If it is formatted by format() in yarn.util.Times, it appears as 10-Apr-2013 16:29:56 Same value, but different timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-570) Time strings are formated in different timezone
[ https://issues.apache.org/jira/browse/YARN-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated YARN-570: --- Attachment: YARN-570.5.patch Time strings are formated in different timezone --- Key: YARN-570 URL: https://issues.apache.org/jira/browse/YARN-570 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.2.0 Reporter: Peng Zhang Assignee: Akira AJISAKA Attachments: MAPREDUCE-5141.patch, YARN-570.2.patch, YARN-570.3.patch, YARN-570.4.patch, YARN-570.5.patch Time strings on different page are displayed in different timezone. If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as Wed, 10 Apr 2013 08:29:56 GMT If it is formatted by format() in yarn.util.Times, it appears as 10-Apr-2013 16:29:56 Same value, but different timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-570) Time strings are formated in different timezone
[
https://issues.apache.org/jira/browse/YARN-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172095#comment-14172095
]
Hadoop QA commented on YARN-570:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674935/YARN-570.5.patch
against trunk revision 128ace1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5403//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5403//console
This message is automatically generated.
Time strings are formated in different timezone
---
Key: YARN-570
URL: https://issues.apache.org/jira/browse/YARN-570
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.2.0
Reporter: Peng Zhang
Assignee: Akira AJISAKA
Attachments: MAPREDUCE-5141.patch, YARN-570.2.patch,
YARN-570.3.patch, YARN-570.4.patch, YARN-570.5.patch
Time strings on different page are displayed in different timezone.
If it is rendered by renderHadoopDate() in yarn.dt.plugins.js, it appears as
Wed, 10 Apr 2013 08:29:56 GMT
If it is formatted by format() in yarn.util.Times, it appears as 10-Apr-2013
16:29:56
Same value, but different timezone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated YARN-2198: --- Attachment: YARN-2198.13.patch .13.patch rebased to current trunk fixes. Remove the need to run NodeManager as privileged account for Windows Secure Container Executor -- Key: YARN-2198 URL: https://issues.apache.org/jira/browse/YARN-2198 Project: Hadoop YARN Issue Type: Improvement Reporter: Remus Rusanu Assignee: Remus Rusanu Labels: security, windows Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch YARN-1972 introduces a Secure Windows Container Executor. However this executor requires the process launching the container to be LocalSystem or a member of the a local Administrators group. Since the process in question is the NodeManager, the requirement translates to the entire NM to run as a privileged account, a very large surface area to review and protect. This proposal is to move the privileged operations into a dedicated NT service. The NM can run as a low privilege account and communicate with the privileged NT service when it needs to launch a container. This would reduce the surface exposed to the high privileges. There has to exist a secure, authenticated and authorized channel of communication between the NM and the privileged NT service. Possible alternatives are a new TCP endpoint, Java RPC etc. My proposal though would be to use Windows LPC (Local Procedure Calls), which is a Windows platform specific inter-process communication channel that satisfies all requirements and is easy to deploy. The privileged NT service would register and listen on an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop with libwinutils which would host the LPC client code. The client would connect to the LPC port (NtConnectPort) and send a message requesting a container launch (NtRequestWaitReplyPort). LPC provides authentication and the privileged NT service can use authorization API (AuthZ) to validate the caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2682) WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir.
[ https://issues.apache.org/jira/browse/YARN-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172125#comment-14172125 ] Remus Rusanu commented on YARN-2682: Thanks [~zxu]! YARN-2198 will probably conflict with this but is only cosmetic. as soon as you commit, I'll resolve the conflicts. WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir. - Key: YARN-2682 URL: https://issues.apache.org/jira/browse/YARN-2682 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Priority: Minor Attachments: YARN-2682.000.patch, YARN-2682.001.patch DefaultContainerExecutor won't use getFirstApplicationDir any more. But we can't delete getFirstApplicationDir in DefaultContainerExecutor because WindowsSecureContainerExecutor uses it. We should move getFirstApplicationDir function from DefaultContainerExecutor to WindowsSecureContainerExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2198) Remove the need to run NodeManager as privileged account for Windows Secure Container Executor
[ https://issues.apache.org/jira/browse/YARN-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172223#comment-14172223 ] Remus Rusanu commented on YARN-2198: The findbugs issues are unrelated to the patch: - Inconsistent synchronization of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.delegationTokenSequenceNumber; locked 71% of time - Dereference of the result of readLine() without nullcheck in org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName() Remove the need to run NodeManager as privileged account for Windows Secure Container Executor -- Key: YARN-2198 URL: https://issues.apache.org/jira/browse/YARN-2198 Project: Hadoop YARN Issue Type: Improvement Reporter: Remus Rusanu Assignee: Remus Rusanu Labels: security, windows Attachments: .YARN-2198.delta.10.patch, YARN-2198.1.patch, YARN-2198.11.patch, YARN-2198.12.patch, YARN-2198.13.patch, YARN-2198.2.patch, YARN-2198.3.patch, YARN-2198.delta.4.patch, YARN-2198.delta.5.patch, YARN-2198.delta.6.patch, YARN-2198.delta.7.patch, YARN-2198.separation.patch, YARN-2198.trunk.10.patch, YARN-2198.trunk.4.patch, YARN-2198.trunk.5.patch, YARN-2198.trunk.6.patch, YARN-2198.trunk.8.patch, YARN-2198.trunk.9.patch YARN-1972 introduces a Secure Windows Container Executor. However this executor requires the process launching the container to be LocalSystem or a member of the a local Administrators group. Since the process in question is the NodeManager, the requirement translates to the entire NM to run as a privileged account, a very large surface area to review and protect. This proposal is to move the privileged operations into a dedicated NT service. The NM can run as a low privilege account and communicate with the privileged NT service when it needs to launch a container. This would reduce the surface exposed to the high privileges. There has to exist a secure, authenticated and authorized channel of communication between the NM and the privileged NT service. Possible alternatives are a new TCP endpoint, Java RPC etc. My proposal though would be to use Windows LPC (Local Procedure Calls), which is a Windows platform specific inter-process communication channel that satisfies all requirements and is easy to deploy. The privileged NT service would register and listen on an LPC port (NtCreatePort, NtListenPort). The NM would use JNI to interop with libwinutils which would host the LPC client code. The client would connect to the LPC port (NtConnectPort) and send a message requesting a container launch (NtRequestWaitReplyPort). LPC provides authentication and the privileged NT service can use authorization API (AuthZ) to validate the caller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2656) RM web services authentication filter should add support for proxy user
[ https://issues.apache.org/jira/browse/YARN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172276#comment-14172276 ] Hudson commented on YARN-2656: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #712 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/712/]) YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. (zjshen: rev 1220bb72d452521c6f09cebe1dd77341054ee9dd) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokenAuthentication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java * hadoop-yarn-project/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java RM web services authentication filter should add support for proxy user --- Key: YARN-2656 URL: https://issues.apache.org/jira/browse/YARN-2656 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Varun Vasudev Assignee: Zhijie Shen Fix For: 2.6.0 Attachments: YARN-2656.3.patch, YARN-2656.4.patch, YARN-2656.5.patch, apache-yarn-2656.0.patch, apache-yarn-2656.1.patch, apache-yarn-2656.2.patch The DelegationTokenAuthenticationFilter adds support for doAs functionality. The RMAuthenticationFilter should expose this as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2588) Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception.
[ https://issues.apache.org/jira/browse/YARN-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated YARN-2588: - Attachment: YARN-2588.1.patch Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception. -- Key: YARN-2588 URL: https://issues.apache.org/jira/browse/YARN-2588 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 2.6.0, 2.5.1 Reporter: Rohith Assignee: Rohith Attachments: YARN-2588.1.patch, YARN-2588.patch Consider scenario where, StandBy RM is failed to transition to Active because of ZK exception(connectionLoss or SessionExpired). Then any further transition to Active for same RM does not move RM to Active state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2588) Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception.
[ https://issues.apache.org/jira/browse/YARN-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172284#comment-14172284 ] Rohith commented on YARN-2588: -- I updated the patch with above changes.Please review Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception. -- Key: YARN-2588 URL: https://issues.apache.org/jira/browse/YARN-2588 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 2.6.0, 2.5.1 Reporter: Rohith Assignee: Rohith Attachments: YARN-2588.1.patch, YARN-2588.patch Consider scenario where, StandBy RM is failed to transition to Active because of ZK exception(connectionLoss or SessionExpired). Then any further transition to Active for same RM does not move RM to Active state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2588) Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception.
[
https://issues.apache.org/jira/browse/YARN-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172335#comment-14172335
]
Hadoop QA commented on YARN-2588:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674976/YARN-2588.1.patch
against trunk revision 128ace1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5405//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5405//console
This message is automatically generated.
Standby RM does not transitionToActive if previous transitionToActive is
failed with ZK exception.
--
Key: YARN-2588
URL: https://issues.apache.org/jira/browse/YARN-2588
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 3.0.0, 2.6.0, 2.5.1
Reporter: Rohith
Assignee: Rohith
Attachments: YARN-2588.1.patch, YARN-2588.patch
Consider scenario where, StandBy RM is failed to transition to Active because
of ZK exception(connectionLoss or SessionExpired). Then any further
transition to Active for same RM does not move RM to Active state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2579) Both RM's state is Active , but 1 RM is not really active.
[ https://issues.apache.org/jira/browse/YARN-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated YARN-2579: - Attachment: YARN-2579.patch Both RM's state is Active , but 1 RM is not really active. -- Key: YARN-2579 URL: https://issues.apache.org/jira/browse/YARN-2579 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.5.1 Reporter: Rohith Assignee: Rohith Attachments: YARN-2579.patch, YARN-2579.patch I encountered a situaltion where both RM's web page was able to access and its state displayed as Active. But One of the RM's ActiveServices were stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2579) Both RM's state is Active , but 1 RM is not really active.
[ https://issues.apache.org/jira/browse/YARN-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172369#comment-14172369 ] Rohith commented on YARN-2579: -- I updated the patch with test that simulates transitionToStandBy causes RM to hang in specific flow of events from ZK. Please review the patch. Both RM's state is Active , but 1 RM is not really active. -- Key: YARN-2579 URL: https://issues.apache.org/jira/browse/YARN-2579 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.5.1 Reporter: Rohith Assignee: Rohith Attachments: YARN-2579.patch, YARN-2579.patch I encountered a situaltion where both RM's web page was able to access and its state displayed as Active. But One of the RM's ActiveServices were stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2656) RM web services authentication filter should add support for proxy user
[ https://issues.apache.org/jira/browse/YARN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172376#comment-14172376 ] Hudson commented on YARN-2656: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1902 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1902/]) YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. (zjshen: rev 1220bb72d452521c6f09cebe1dd77341054ee9dd) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokenAuthentication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java * hadoop-yarn-project/CHANGES.txt RM web services authentication filter should add support for proxy user --- Key: YARN-2656 URL: https://issues.apache.org/jira/browse/YARN-2656 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Varun Vasudev Assignee: Zhijie Shen Fix For: 2.6.0 Attachments: YARN-2656.3.patch, YARN-2656.4.patch, YARN-2656.5.patch, apache-yarn-2656.0.patch, apache-yarn-2656.1.patch, apache-yarn-2656.2.patch The DelegationTokenAuthenticationFilter adds support for doAs functionality. The RMAuthenticationFilter should expose this as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster
[
https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172419#comment-14172419
]
Hadoop QA commented on YARN-2314:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674827/YARN-2314.patch
against trunk revision 128ace1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5408//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5408//console
This message is automatically generated.
ContainerManagementProtocolProxy can create thousands of threads for a large
cluster
Key: YARN-2314
URL: https://issues.apache.org/jira/browse/YARN-2314
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: YARN-2314.patch, disable-cm-proxy-cache.patch,
nmproxycachefix.prototype.patch
ContainerManagementProtocolProxy has a cache of NM proxies, and the size of
this cache is configurable. However the cache can grow far beyond the
configured size when running on a large cluster and blow AM address/container
limits. More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2690) Make ReservationSystem and its dependent classes independent of Scheduler type
Anubhav Dhoot created YARN-2690: --- Summary: Make ReservationSystem and its dependent classes independent of Scheduler type Key: YARN-2690 URL: https://issues.apache.org/jira/browse/YARN-2690 Project: Hadoop YARN Issue Type: Sub-task Reporter: Anubhav Dhoot A lot of common reservation classes depend on CapacityScheduler and specifically its configuration. This jira is to make them ready for other Schedulers by abstracting out the configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster
[
https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172422#comment-14172422
]
Hadoop QA commented on YARN-2314:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674827/YARN-2314.patch
against trunk revision 128ace1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5407//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5407//console
This message is automatically generated.
ContainerManagementProtocolProxy can create thousands of threads for a large
cluster
Key: YARN-2314
URL: https://issues.apache.org/jira/browse/YARN-2314
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: YARN-2314.patch, disable-cm-proxy-cache.patch,
nmproxycachefix.prototype.patch
ContainerManagementProtocolProxy has a cache of NM proxies, and the size of
this cache is configurable. However the cache can grow far beyond the
configured size when running on a large cluster and blow AM address/container
limits. More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2690) Make ReservationSystem and its dependent classes independent of Scheduler type
[ https://issues.apache.org/jira/browse/YARN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot reassigned YARN-2690: --- Assignee: Anubhav Dhoot Make ReservationSystem and its dependent classes independent of Scheduler type Key: YARN-2690 URL: https://issues.apache.org/jira/browse/YARN-2690 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot A lot of common reservation classes depend on CapacityScheduler and specifically its configuration. This jira is to make them ready for other Schedulers by abstracting out the configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2579) Both RM's state is Active , but 1 RM is not really active.
[
https://issues.apache.org/jira/browse/YARN-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172442#comment-14172442
]
Hadoop QA commented on YARN-2579:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674998/YARN-2579.patch
against trunk revision 128ace1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1267 javac
compiler warnings (more than the trunk's current 1266 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5406//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5406//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5406//console
This message is automatically generated.
Both RM's state is Active , but 1 RM is not really active.
--
Key: YARN-2579
URL: https://issues.apache.org/jira/browse/YARN-2579
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.5.1
Reporter: Rohith
Assignee: Rohith
Attachments: YARN-2579.patch, YARN-2579.patch
I encountered a situaltion where both RM's web page was able to access and
its state displayed as Active. But One of the RM's ActiveServices were
stopped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2656) RM web services authentication filter should add support for proxy user
[ https://issues.apache.org/jira/browse/YARN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172454#comment-14172454 ] Hudson commented on YARN-2656: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1927 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1927/]) YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. (zjshen: rev 1220bb72d452521c6f09cebe1dd77341054ee9dd) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokenAuthentication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java RM web services authentication filter should add support for proxy user --- Key: YARN-2656 URL: https://issues.apache.org/jira/browse/YARN-2656 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Varun Vasudev Assignee: Zhijie Shen Fix For: 2.6.0 Attachments: YARN-2656.3.patch, YARN-2656.4.patch, YARN-2656.5.patch, apache-yarn-2656.0.patch, apache-yarn-2656.1.patch, apache-yarn-2656.2.patch The DelegationTokenAuthenticationFilter adds support for doAs functionality. The RMAuthenticationFilter should expose this as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: (was: YARN-2280.patch) Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Trivial Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2280) Resource manager web service fields are not accessible
[ https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated YARN-2280: Attachment: YARN-2280.patch Resource manager web service fields are not accessible -- Key: YARN-2280 URL: https://issues.apache.org/jira/browse/YARN-2280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0, 2.4.1 Reporter: Krisztian Horvath Assignee: Krisztian Horvath Priority: Trivial Fix For: 2.6.0 Attachments: YARN-2280.patch Using the resource manager's rest api (org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some rest call returns a class where the fields after the unmarshal cannot be accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same classes on client side these fields only accessible via reflection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster
[ https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172497#comment-14172497 ] Wangda Tan commented on YARN-2314: -- [~jlowe], Thanks for updating the patch, generally it looks very good to me. A minor suggestion is, I'd suggest to change yarn.client.max-nodemanagers-proxies to yarn.client.max-cached-nodemanagers-proxies. Because when the user set the MAX_NM_PROXIES to 0, he/she can still connect to more than 0 NMs, even if the java docs is very clear, but the name is still a little confusing. Wangda ContainerManagementProtocolProxy can create thousands of threads for a large cluster Key: YARN-2314 URL: https://issues.apache.org/jira/browse/YARN-2314 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: YARN-2314.patch, disable-cm-proxy-cache.patch, nmproxycachefix.prototype.patch ContainerManagementProtocolProxy has a cache of NM proxies, and the size of this cache is configurable. However the cache can grow far beyond the configured size when running on a large cluster and blow AM address/container limits. More details in the first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2280) Resource manager web service fields are not accessible
[
https://issues.apache.org/jira/browse/YARN-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172564#comment-14172564
]
Hadoop QA commented on YARN-2280:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675016/YARN-2280.patch
against trunk revision 128ace1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5409//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5409//console
This message is automatically generated.
Resource manager web service fields are not accessible
--
Key: YARN-2280
URL: https://issues.apache.org/jira/browse/YARN-2280
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.4.0, 2.4.1
Reporter: Krisztian Horvath
Assignee: Krisztian Horvath
Priority: Trivial
Fix For: 2.6.0
Attachments: YARN-2280.patch
Using the resource manager's rest api
(org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices) some
rest call returns a class where the fields after the unmarshal cannot be
accessible. For example SchedulerTypeInfo - schedulerInfo. Using the same
classes on client side these fields only accessible via reflection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-2496:
-
Attachment: YARN-2496-20141015-1.patch
[~vinodkv], Thanks for your comments!
Updated patch addressed all your comments above, will separated JIRAs as what
you suggested to solve other problems right now.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496-20141009-1.patch, YARN-2496-20141014-1.patch,
YARN-2496-20141015-1.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-2496:
-
Attachment: YARN-2496+YARN-2500.patch
Since YARN-2496 and YARN-2500 need apply together to make it compilable. So
updated a patch contains YARN-2496 and YARN-2500 to get Jenkins report.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496-20141009-1.patch,
YARN-2496-20141014-1.patch, YARN-2496-20141015-1.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2689) TestSecureRMRegistryOperations failing on windows
[
https://issues.apache.org/jira/browse/YARN-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172629#comment-14172629
]
Chris Nauroth commented on YARN-2689:
-
Hi Steve. Thanks for fixing this. The patch looks good to me, and I verified
that it fixes the tests on Windows. However, I'm seeing a problem when running
on Mac and Linux. It's hanging while executing {{ktutil}}. On my systems,
{{ktutil}} is an interactive command, so the tests are starting up the child
process, and then it's never exiting. (See stack trace below.) Some quick
searching indicates that some installations of {{ktutil}} are non-interactive,
but others are entirely interactive (MIT for example).
{code}
JUnit prio=10 tid=0x7f424488c000 nid=0x675 runnable [0x7f4239c02000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:236)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked 0xed550918 (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:282)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:176)
- locked 0xed3df918 (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:153)
at java.io.BufferedReader.read1(BufferedReader.java:204)
at java.io.BufferedReader.read(BufferedReader.java:278)
- locked 0xed3df918 (a java.io.InputStreamReader)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:721)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:530)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:708)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:797)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:780)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
TestSecureRMRegistryOperations failing on windows
-
Key: YARN-2689
URL: https://issues.apache.org/jira/browse/YARN-2689
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, resourcemanager
Affects Versions: 2.6.0
Environment: Windows server, Java 7, ZK 3.4.6
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-2689-001.patch
the micro ZK service used in the {{TestSecureRMRegistryOperations}} test
doesnt start on windows,
{code}
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Could
not configure server because SASL configuration did not allow the ZooKeeper
server to authenticate itself properly:
javax.security.auth.login.LoginException: Unable to obtain password from user
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-1963: -- Attachment: (was: YARN Application Priorities Design.pdf) Support priorities across applications within the same queue - Key: YARN-1963 URL: https://issues.apache.org/jira/browse/YARN-1963 Project: Hadoop YARN Issue Type: New Feature Components: api, resourcemanager Reporter: Arun C Murthy Assignee: Sunil G It will be very useful to support priorities among applications within the same queue, particularly in production scenarios. It allows for finer-grained controls without having to force admins to create a multitude of queues, plus allows existing applications to continue using existing queues which are usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-1963: -- Attachment: YARN Application Priorities Design.pdf Updating document with added information about user/admin level api information. Also creating initial set of jiras for user/admin apis and priority-label manager to manage priority labels in RM. Support priorities across applications within the same queue - Key: YARN-1963 URL: https://issues.apache.org/jira/browse/YARN-1963 Project: Hadoop YARN Issue Type: New Feature Components: api, resourcemanager Reporter: Arun C Murthy Assignee: Sunil G Attachments: YARN Application Priorities Design.pdf It will be very useful to support priorities among applications within the same queue, particularly in production scenarios. It allows for finer-grained controls without having to force admins to create a multitude of queues, plus allows existing applications to continue using existing queues which are usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2689) TestSecureRMRegistryOperations failing on windows
[
https://issues.apache.org/jira/browse/YARN-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172656#comment-14172656
]
Steve Loughran commented on YARN-2689:
--
I'll make a separate issue for that
TestSecureRMRegistryOperations failing on windows
-
Key: YARN-2689
URL: https://issues.apache.org/jira/browse/YARN-2689
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, resourcemanager
Affects Versions: 2.6.0
Environment: Windows server, Java 7, ZK 3.4.6
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-2689-001.patch
the micro ZK service used in the {{TestSecureRMRegistryOperations}} test
doesnt start on windows,
{code}
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Could
not configure server because SASL configuration did not allow the ZooKeeper
server to authenticate itself properly:
javax.security.auth.login.LoginException: Unable to obtain password from user
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2691) User level API support for priority label
Sunil G created YARN-2691: - Summary: User level API support for priority label Key: YARN-2691 URL: https://issues.apache.org/jira/browse/YARN-2691 Project: Hadoop YARN Issue Type: Sub-task Components: client Reporter: Sunil G Assignee: Rohith Support for handling Application-Priority label coming from client to ApplicationSubmissionContext. Common api support for user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2692) ktutil test hanging on some machines/ktutil versions
Steve Loughran created YARN-2692:
Summary: ktutil test hanging on some machines/ktutil versions
Key: YARN-2692
URL: https://issues.apache.org/jira/browse/YARN-2692
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
a couple of the registry security tests run native {{ktutil}}; this is
primarily to debug the keytab generation. [~cnauroth] reports that some
versions of {{kinit}} hang. Fix: rm the tests. [YARN-2689]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2693) Priority Label Manager in RM to manage priority labels
Sunil G created YARN-2693: - Summary: Priority Label Manager in RM to manage priority labels Key: YARN-2693 URL: https://issues.apache.org/jira/browse/YARN-2693 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Focus of this JIRA to have a centralized service to handle priority labels. Support operations such as * Add/Delete priority label to a specified queue * Manage integer mapping associated with each priority label * Support managing default priority label of a given queue * ACL support in queue level for priority label * Expose interface to RM to validate priority label Storage for this labels will be done in FileSystem and in Memory similar to NodeLabel * FileSystem Based : persistent across RM restart * Memory Based: non-persistent across RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2694) Ensure only single node labels specified in resource request, and node label expression only specified when resourceName=ANY
Wangda Tan created YARN-2694: Summary: Ensure only single node labels specified in resource request, and node label expression only specified when resourceName=ANY Key: YARN-2694 URL: https://issues.apache.org/jira/browse/YARN-2694 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler, resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Currently, node label expression supporting in capacity scheduler is partial completed. Now node label expression specified in Resource Request will only respected when it specified at ANY level. And a ResourceRequest with multiple node labels will make user limit computation becomes tricky. Now we need temporarily disable them, changes include, - AMRMClient - ApplicationMasterService -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2689) TestSecureRMRegistryOperations failing on windows
[
https://issues.apache.org/jira/browse/YARN-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172676#comment-14172676
]
Chris Nauroth commented on YARN-2689:
-
+1 for the patch. Thanks again, Steve.
TestSecureRMRegistryOperations failing on windows
-
Key: YARN-2689
URL: https://issues.apache.org/jira/browse/YARN-2689
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, resourcemanager
Affects Versions: 2.6.0
Environment: Windows server, Java 7, ZK 3.4.6
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-2689-001.patch
the micro ZK service used in the {{TestSecureRMRegistryOperations}} test
doesnt start on windows,
{code}
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Could
not configure server because SASL configuration did not allow the ZooKeeper
server to authenticate itself properly:
javax.security.auth.login.LoginException: Unable to obtain password from user
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2693) Priority Label Manager in RM to manage priority labels
[ https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2693: -- Description: Focus of this JIRA is to have a centralized service to handle priority labels. Support operations such as * Add/Delete priority label to a specified queue * Manage integer mapping associated with each priority label * Support managing default priority label of a given queue * ACL support in queue level for priority label * Expose interface to RM to validate priority label Storage for this labels will be done in FileSystem and in Memory similar to NodeLabel * FileSystem Based : persistent across RM restart * Memory Based: non-persistent across RM restart was: Focus of this JIRA to have a centralized service to handle priority labels. Support operations such as * Add/Delete priority label to a specified queue * Manage integer mapping associated with each priority label * Support managing default priority label of a given queue * ACL support in queue level for priority label * Expose interface to RM to validate priority label Storage for this labels will be done in FileSystem and in Memory similar to NodeLabel * FileSystem Based : persistent across RM restart * Memory Based: non-persistent across RM restart Priority Label Manager in RM to manage priority labels -- Key: YARN-2693 URL: https://issues.apache.org/jira/browse/YARN-2693 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Focus of this JIRA is to have a centralized service to handle priority labels. Support operations such as * Add/Delete priority label to a specified queue * Manage integer mapping associated with each priority label * Support managing default priority label of a given queue * ACL support in queue level for priority label * Expose interface to RM to validate priority label Storage for this labels will be done in FileSystem and in Memory similar to NodeLabel * FileSystem Based : persistent across RM restart * Memory Based: non-persistent across RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2695) Support continuously looking reserved container with node labels
Wangda Tan created YARN-2695: Summary: Support continuously looking reserved container with node labels Key: YARN-2695 URL: https://issues.apache.org/jira/browse/YARN-2695 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler, resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan YARN-1769 improved capacity scheduler to continuously look at reserved container when trying to reserve/allocate resource. This should be respected when node/resource-request/queue has node label. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2696) Queue sorting in CapacityScheduler should consider node label
Wangda Tan created YARN-2696: Summary: Queue sorting in CapacityScheduler should consider node label Key: YARN-2696 URL: https://issues.apache.org/jira/browse/YARN-2696 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler, resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan In the past, when trying to allocate containers under a parent queue in CapacityScheduler. The parent queue will choose child queues by the used resource from smallest to largest. Now we support node label in CapacityScheduler, we should also consider used resource in child queues by node labels when allocating resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172703#comment-14172703
]
Wangda Tan commented on YARN-2496:
--
Filed YARN-2694 for disable multiple node labels in resource request, and check
node label expression only specified in ANY level.
Filed YARN-2695 for respecting continuos container reservation with node label
Filed YARN-2696 for considering node labels when sorting queues.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496-20141009-1.patch,
YARN-2496-20141014-1.patch, YARN-2496-20141015-1.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2504) Support get/add/remove/change labels in RM admin CLI
[ https://issues.apache.org/jira/browse/YARN-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172717#comment-14172717 ] Sunil G commented on YARN-2504: --- One comment: How *response* is populated for addLabels/removeLabels etc in AdminService.java. I think the validation should happen in label manager and manager should populate the same. Support get/add/remove/change labels in RM admin CLI - Key: YARN-2504 URL: https://issues.apache.org/jira/browse/YARN-2504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2504.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2641) Decommission nodes on -refreshNodes instead of next NM-RM heartbeat
[ https://issues.apache.org/jira/browse/YARN-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172718#comment-14172718 ] zhihai xu commented on YARN-2641: - thanks [~kasha], [~jianhe], [~djp] and [~ywskycn] for the review. Decommission nodes on -refreshNodes instead of next NM-RM heartbeat --- Key: YARN-2641 URL: https://issues.apache.org/jira/browse/YARN-2641 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: YARN-2641.000.patch, YARN-2641.001.patch, YARN-2641.002.patch, YARN-2641.003.patch improve node decommission latency in RM. Currently the node decommission only happened after RM received nodeHeartbeat from the Node Manager. The node heartbeat interval is configurable. The default value is 1 second. It will be better to do the decommission during RM Refresh(NodesListManager) instead of nodeHeartbeat(ResourceTrackerService). This will be a much more serious issue: After RM is refreshed (refreshNodes), If the NM to be decommissioned is killed before NM sent heartbeat to RM. The RMNode will never be decommissioned in RM. The RMNode will only expire in RM after yarn.nm.liveness-monitor.expiry-interval-ms(default value 10 minutes) time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2692) ktutil test hanging on some machines/ktutil versions
[
https://issues.apache.org/jira/browse/YARN-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-2692:
-
Component/s: (was: api)
(was: resourcemanager)
ktutil test hanging on some machines/ktutil versions
Key: YARN-2692
URL: https://issues.apache.org/jira/browse/YARN-2692
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-2692-001.patch
a couple of the registry security tests run native {{ktutil}}; this is
primarily to debug the keytab generation. [~cnauroth] reports that some
versions of {{kinit}} hang. Fix: rm the tests. [YARN-2689]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2566) DefaultContainerExecutor should pick a working directory randomly
[
https://issues.apache.org/jira/browse/YARN-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172725#comment-14172725
]
zhihai xu commented on YARN-2566:
-
thanks [~kasha], [~vinodkv] and [~vvasudev] for the review.
DefaultContainerExecutor should pick a working directory randomly
-
Key: YARN-2566
URL: https://issues.apache.org/jira/browse/YARN-2566
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.5.0
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Critical
Fix For: 2.6.0
Attachments: YARN-2566.000.patch, YARN-2566.001.patch,
YARN-2566.002.patch, YARN-2566.003.patch, YARN-2566.004.patch,
YARN-2566.005.patch, YARN-2566.006.patch, YARN-2566.007.patch,
YARN-2566.008.patch
startLocalizer in DefaultContainerExecutor will only use the first localDir
to copy the token file, if the copy is failed for first localDir due to not
enough disk space in the first localDir, the localization will be failed even
there are plenty of disk space in other localDirs. We see the following error
for this case:
{code}
2014-09-13 23:33:25,171 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Unable to
create app directory
/hadoop/d1/usercache/cloudera/appcache/application_1410663092546_0004
java.io.IOException: mkdir of
/hadoop/d1/usercache/cloudera/appcache/application_1410663092546_0004 failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1062)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:157)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:721)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:717)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:717)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:426)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppDirs(DefaultContainerExecutor.java:522)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:94)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:987)
2014-09-13 23:33:25,185 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.FileNotFoundException: File
file:/hadoop/d1/usercache/cloudera/appcache/application_1410663092546_0004
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
at
org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:111)
at
org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:76)
at
org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.init(ChecksumFs.java:344)
at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:390)
at
org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:677)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:673)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.create(FileContext.java:673)
at org.apache.hadoop.fs.FileContext$Util.copy(FileContext.java:2021)
at org.apache.hadoop.fs.FileContext$Util.copy(FileContext.java:1963)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:102)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:987)
2014-09-13 23:33:25,186 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1410663092546_0004_01_01 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2014-09-13 23:33:25,187 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=cloudera
OPERATION=Container Finished - Failed TARGET=ContainerImpl
RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED
[jira] [Updated] (YARN-2692) ktutil test hanging on some machines/ktutil versions
[
https://issues.apache.org/jira/browse/YARN-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-2692:
-
Attachment: YARN-2692-001.patch
patch -001 simply removes the ktutil test, which was only there to debug minikdc
ktutil test hanging on some machines/ktutil versions
Key: YARN-2692
URL: https://issues.apache.org/jira/browse/YARN-2692
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-2692-001.patch
a couple of the registry security tests run native {{ktutil}}; this is
primarily to debug the keytab generation. [~cnauroth] reports that some
versions of {{kinit}} hang. Fix: rm the tests. [YARN-2689]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172741#comment-14172741
]
Hadoop QA commented on YARN-2496:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675049/YARN-2496%2BYARN-2500.patch
against trunk revision b9edad6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage
The following test timeouts occurred in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5410//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5410//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5410//console
This message is automatically generated.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496-20141009-1.patch,
YARN-2496-20141014-1.patch, YARN-2496-20141015-1.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2682) WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir.
[ https://issues.apache.org/jira/browse/YARN-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172747#comment-14172747 ] zhihai xu commented on YARN-2682: - Hi [~jianhe], I saw you are reviewing YARN-2198, As [~rusanu] said, we need review and commit this JIRA before YARN-2198 , so remus can resolve the conflict in YARN-2198. Do you have time to review this JIRA? thanks WindowsSecureContainerExecutor should not depend on DefaultContainerExecutor#getFirstApplicationDir. - Key: YARN-2682 URL: https://issues.apache.org/jira/browse/YARN-2682 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.5.0 Reporter: zhihai xu Assignee: zhihai xu Priority: Minor Attachments: YARN-2682.000.patch, YARN-2682.001.patch DefaultContainerExecutor won't use getFirstApplicationDir any more. But we can't delete getFirstApplicationDir in DefaultContainerExecutor because WindowsSecureContainerExecutor uses it. We should move getFirstApplicationDir function from DefaultContainerExecutor to WindowsSecureContainerExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2505) Support get/add/remove/change labels in RM REST API
[
https://issues.apache.org/jira/browse/YARN-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172769#comment-14172769
]
Vinod Kumar Vavilapalli commented on YARN-2505:
---
bq. Thanks for update. However, I just compared two patches, it seems the new
one is double-sized of the old one.
Yes. Many changes are triplicated. For e.g.
ApplicationSubmissionContextInfo.java
Some suggestions for renames
- LabelInfo - NodeLabelInfo. Similarly the web-service response tags in XML
and Json.
- LabelNamesInfo - ClusterNodeLabelsInfo. Similarly labelNamesInfo -
cluster-node-labels-info
- Not sure what LabelsInfo is useful fo
- NodeInfo.labels - nodeLabels. Similarly the getter.
h3. APIs
Here's a modified proposal for the APIs, given
- all the renames we have done in other patches
- existing APIs like /nodes/nodeID
h4. Cluster Node-labels APIs
||Request||API||JSON Response||Authorization|
|List cluster node-labels|GET /cluster-node-labels/|[X, Y, Z]|All users|
|Create new cluster node-labels|PUT /cluster-node-labels/comma-separated list
of labels (We can also replace this with or additionally support a POST with
an array as the body)|{}|Admins|
|Remove some cluster-node-labels|DELETE /cluster-node-labels/comma-separated
list of labels|{}|Admins|
h4. APIs for node-to-label mappings
||Request||API||JSON Response||Authorization|
|List all labels for a node|GET /nodes/nodeId/node-labels|[X, Y, Z]|All users|
|Add a bunch of node-label to a node|PUT
/nodes/nodeId/node-labels/comma-separated list of labels (We can also
replace this with or additionally support a post with an array as the
body)|{}|Admins|
|Remove a label from a node|DELETE /nodes/nodeId/node-labels/comma-separated
list of labels|{}|Admins|
The queue API is missing overall from RMWeb-services - this is not RESTified
yet. We can track that in its own JIRA. There are more APIs that are
good-to-have and that I think should be added separately.
What do you think, [~leftnoteasy]/[~ksumit]/[~cwelch]?
Support get/add/remove/change labels in RM REST API
---
Key: YARN-2505
URL: https://issues.apache.org/jira/browse/YARN-2505
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Craig Welch
Attachments: YARN-2505.1.patch, YARN-2505.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2692) ktutil test hanging on some machines/ktutil versions
[
https://issues.apache.org/jira/browse/YARN-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172787#comment-14172787
]
Hadoop QA commented on YARN-2692:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675073/YARN-2692-001.patch
against trunk revision f19771a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5411//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5411//console
This message is automatically generated.
ktutil test hanging on some machines/ktutil versions
Key: YARN-2692
URL: https://issues.apache.org/jira/browse/YARN-2692
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-2692-001.patch
a couple of the registry security tests run native {{ktutil}}; this is
primarily to debug the keytab generation. [~cnauroth] reports that some
versions of {{kinit}} hang. Fix: rm the tests. [YARN-2689]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2505) Support get/add/remove/change labels in RM REST API
[ https://issues.apache.org/jira/browse/YARN-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172792#comment-14172792 ] Sumit Kumar commented on YARN-2505: --- # /nodes/nodeId/node-labels seems like an over-use of node before labels because the prefixes in the REST url kind of seem deterministic of the context. should we replace that with just labels instead? # same for /cluster-node-labels, what do you think about /cluster/labels? ## do we foresee a usecase where labels could be provided on resources/entities other than nodes? for example applications? ## if the intention is to highlight that these labels are only for nodes, do you think passing a parameter type=node and likewise would make more sense? this of course would be useful only when the above question holds true. Support get/add/remove/change labels in RM REST API --- Key: YARN-2505 URL: https://issues.apache.org/jira/browse/YARN-2505 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Craig Welch Attachments: YARN-2505.1.patch, YARN-2505.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2505) Support get/add/remove/change labels in RM REST API
[ https://issues.apache.org/jira/browse/YARN-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172807#comment-14172807 ] Vinod Kumar Vavilapalli commented on YARN-2505: --- bq. /nodes/nodeId/node-labels seems like an over-use of node before labels because the prefixes in the REST url kind of seem deterministic of the context. should we replace that with just labels instead? +1 bq. same for /cluster-node-labels, what do you think about /cluster/labels? +1, I like this much better, but /cluster/node-labels instead. bq. do we foresee a usecase where labels could be provided on resources/entities other than nodes? for example applications? Yeah, we already have known priorities/priority labels being proposed at YARN-1963, so I made sure we explicitly say node-labels everywhere to avoid confusion. bq. if the intention is to highlight that these labels are only for nodes, do you think passing a parameter type=node and likewise would make more sense? this of course would be useful only when the above question holds true. We will then have to invent a label-type, seems like an over-kill to me. Support get/add/remove/change labels in RM REST API --- Key: YARN-2505 URL: https://issues.apache.org/jira/browse/YARN-2505 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Craig Welch Attachments: YARN-2505.1.patch, YARN-2505.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2690) Make ReservationSystem and its dependent classes independent of Scheduler type
[ https://issues.apache.org/jira/browse/YARN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated YARN-2690: Attachment: YARN-2690.001.patch Make ReservationSystem and its dependent classes independent of Scheduler type Key: YARN-2690 URL: https://issues.apache.org/jira/browse/YARN-2690 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2690.001.patch A lot of common reservation classes depend on CapacityScheduler and specifically its configuration. This jira is to make them ready for other Schedulers by abstracting out the configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2505) Support get/add/remove/change labels in RM REST API
[ https://issues.apache.org/jira/browse/YARN-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172856#comment-14172856 ] Sumit Kumar commented on YARN-2505: --- bq. We will then have to invent a label-type, seems like an over-kill to me. Agreed but in light of YARN-1963, would that be useful? So we've come across two label types so far one for nodes and the other for applications? If it helps to do this in context of the other JIRA, may be we should add a new task for the same. Thoughts? Support get/add/remove/change labels in RM REST API --- Key: YARN-2505 URL: https://issues.apache.org/jira/browse/YARN-2505 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Craig Welch Attachments: YARN-2505.1.patch, YARN-2505.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2312) Marking ContainerId#getId as deprecated
[
https://issues.apache.org/jira/browse/YARN-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-2312:
-
Attachment: YARN-2312-branch-2.8.patch
[~jianhe] [~jlowe] Thanks for your comment. Attaching a patch for branch-2.
Marking ContainerId#getId as deprecated
---
Key: YARN-2312
URL: https://issues.apache.org/jira/browse/YARN-2312
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2312-branch-2.8.patch, YARN-2312-wip.patch,
YARN-2312.1.patch, YARN-2312.2-2.patch, YARN-2312.2-3.patch,
YARN-2312.2.patch, YARN-2312.4.patch, YARN-2312.5.patch, YARN-2312.6.patch,
YARN-2312.7.patch
{{ContainerId#getId}} will only return partial value of containerId, only
sequence number of container id without epoch, after YARN-2229. We should
mark {{ContainerId#getId}} as deprecated and use
{{ContainerId#getContainerId}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2690) Make ReservationSystem and its dependent classes independent of Scheduler type
[
https://issues.apache.org/jira/browse/YARN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172915#comment-14172915
]
Hadoop QA commented on YARN-2690:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675091/YARN-2690.001.patch
against trunk revision f19771a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5412//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5412//console
This message is automatically generated.
Make ReservationSystem and its dependent classes independent of Scheduler
type
Key: YARN-2690
URL: https://issues.apache.org/jira/browse/YARN-2690
Project: Hadoop YARN
Issue Type: Sub-task
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Attachments: YARN-2690.001.patch
A lot of common reservation classes depend on CapacityScheduler and
specifically its configuration. This jira is to make them ready for other
Schedulers by abstracting out the configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated YARN-2556: -- Attachment: YARN-2556-WIP.patch A working in progress patch. Any reply will be appreciated. Tool to measure the performance of the timeline server -- Key: YARN-2556 URL: https://issues.apache.org/jira/browse/YARN-2556 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: chang li Attachments: YARN-2556-WIP.patch We need to be able to understand the capacity model for the timeline server to give users the tools they need to deploy a timeline server with the correct capacity. I propose we create a mapreduce job that can measure timeline server write and read performance. Transactions per second, I/O for both read and write would be a good start. This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster
[ https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-2314: - Attachment: YARN-2314v2.patch Updated the patch to deprecate yarn.client.max-nodemanagers-proxies in favor of yarn.client.max-cached-nodemanagers-proxies. ContainerManagementProtocolProxy can create thousands of threads for a large cluster Key: YARN-2314 URL: https://issues.apache.org/jira/browse/YARN-2314 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: YARN-2314.patch, YARN-2314v2.patch, disable-cm-proxy-cache.patch, nmproxycachefix.prototype.patch ContainerManagementProtocolProxy has a cache of NM proxies, and the size of this cache is configurable. However the cache can grow far beyond the configured size when running on a large cluster and blow AM address/container limits. More details in the first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-2496:
-
Attachment: YARN-2496-20141015-2.patch
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496-20141009-1.patch,
YARN-2496-20141014-1.patch, YARN-2496-20141015-1.patch,
YARN-2496-20141015-2.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2500) Miscellaneous changes in ResourceManager to support labels
[ https://issues.apache.org/jira/browse/YARN-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2500: - Attachment: YARN-2500-20141015-1.patch Miscellaneous changes in ResourceManager to support labels -- Key: YARN-2500 URL: https://issues.apache.org/jira/browse/YARN-2500 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2500-20141009-1.patch, YARN-2500-20141009-2.patch, YARN-2500-20141010-1.patch, YARN-2500-20141014-1.patch, YARN-2500-20141015-1.patch, YARN-2500.patch, YARN-2500.patch, YARN-2500.patch, YARN-2500.patch, YARN-2500.patch This patches contains changes in ResourceManager to support labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-2496:
-
Attachment: YARN-2496+YARN-2500.patch
Attached a new patch contains YARN-2496 and YARN-2500 to kick jenkins to get
result.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496+YARN-2500.patch,
YARN-2496-20141009-1.patch, YARN-2496-20141014-1.patch,
YARN-2496-20141015-1.patch, YARN-2496-20141015-2.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2314) ContainerManagementProtocolProxy can create thousands of threads for a large cluster
[
https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173039#comment-14173039
]
Hadoop QA commented on YARN-2314:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675108/YARN-2314v2.patch
against trunk revision f19771a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5414//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5414//console
This message is automatically generated.
ContainerManagementProtocolProxy can create thousands of threads for a large
cluster
Key: YARN-2314
URL: https://issues.apache.org/jira/browse/YARN-2314
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: YARN-2314.patch, YARN-2314v2.patch,
disable-cm-proxy-cache.patch, nmproxycachefix.prototype.patch
ContainerManagementProtocolProxy has a cache of NM proxies, and the size of
this cache is configurable. However the cache can grow far beyond the
configured size when running on a large cluster and blow AM address/container
limits. More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2697) RMAuthenticationHandler is no longer useful
Zhijie Shen created YARN-2697: - Summary: RMAuthenticationHandler is no longer useful Key: YARN-2697 URL: https://issues.apache.org/jira/browse/YARN-2697 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Zhijie Shen After YARN-2656, RMAuthenticationHandler is no longer useful, because authentication mechanism is reusing the common DT auth filter stack. It should be safe to remove this unused code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2312) Marking ContainerId#getId as deprecated
[
https://issues.apache.org/jira/browse/YARN-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173056#comment-14173056
]
Jian He commented on YARN-2312:
---
+1, committing
Marking ContainerId#getId as deprecated
---
Key: YARN-2312
URL: https://issues.apache.org/jira/browse/YARN-2312
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2312-branch-2.8.patch, YARN-2312-wip.patch,
YARN-2312.1.patch, YARN-2312.2-2.patch, YARN-2312.2-3.patch,
YARN-2312.2.patch, YARN-2312.4.patch, YARN-2312.5.patch, YARN-2312.6.patch,
YARN-2312.7.patch
{{ContainerId#getId}} will only return partial value of containerId, only
sequence number of container id without epoch, after YARN-2229. We should
mark {{ContainerId#getId}} as deprecated and use
{{ContainerId#getContainerId}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2312) Marking ContainerId#getId as deprecated
[ https://issues.apache.org/jira/browse/YARN-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173076#comment-14173076 ] Hudson commented on YARN-2312: -- FAILURE: Integrated in Hadoop-trunk-Commit #6267 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6267/]) YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA (jianhe: rev 0af1a2b5bc1469ba22edb63cd58f9b436b1dc4d3) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedJvmID.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalizedResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerIdPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestNMClientAsync.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestCheckpointPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JVMId.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestMemoryApplicationHistoryStore.java Marking ContainerId#getId as deprecated --- Key: YARN-2312 URL: https://issues.apache.org/jira/browse/YARN-2312 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: YARN-2312-branch-2.8.patch, YARN-2312-wip.patch, YARN-2312.1.patch,
[jira] [Commented] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173094#comment-14173094
]
Hadoop QA commented on YARN-2496:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675117/YARN-2496%2BYARN-2500.patch
against trunk revision f19771a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 29 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5415//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5415//console
This message is automatically generated.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496+YARN-2500.patch,
YARN-2496-20141009-1.patch, YARN-2496-20141014-1.patch,
YARN-2496-20141015-1.patch, YARN-2496-20141015-2.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2676) Timeline authentication filter should add support for proxy user
[ https://issues.apache.org/jira/browse/YARN-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2676: -- Attachment: YARN-2676.1.patch Upload a working in progress patch: 1. Server side: make TimelineAuthenticationFilter extend the common DelegationTokenAuthenticationFilter and use the related common stuff. 2. Client side: make TimelineClientImpl use the DelegationTokenAuthenticatedURL and DelegationTokenAuthenticator, which will fail back to pseduo/kerberos authenticator if DT is not there. 3. Client side: make TimelineClientImpl be friendly to proxy user. Will execute the http request with real user with the proxy user set as the doAs user. 4. Cleanup the unnecessary code, which we used to duplicate for timeline security. It is worth mentioning that new http authentication request and response is not going to be compatible with the prior one. It's difficult to be compatible at http level, because both header and body are almost different in format. However, at the point of view of timeline client, the change should be transparent. Timeline authentication filter should add support for proxy user Key: YARN-2676 URL: https://issues.apache.org/jira/browse/YARN-2676 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: YARN-2676.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2676) Timeline authentication filter should add support for proxy user
[ https://issues.apache.org/jira/browse/YARN-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173101#comment-14173101 ] Zhijie Shen commented on YARN-2676: --- The patch needs to base on the one in HADOOP-11207 Timeline authentication filter should add support for proxy user Key: YARN-2676 URL: https://issues.apache.org/jira/browse/YARN-2676 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: YARN-2676.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2312) Marking ContainerId#getId as deprecated
[
https://issues.apache.org/jira/browse/YARN-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173152#comment-14173152
]
Hadoop QA commented on YARN-2312:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675095/YARN-2312-branch-2.8.patch
against trunk revision f19771a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1267 javac
compiler warnings (more than the trunk's current 1266 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5413//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5413//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5413//console
This message is automatically generated.
Marking ContainerId#getId as deprecated
---
Key: YARN-2312
URL: https://issues.apache.org/jira/browse/YARN-2312
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.6.0
Attachments: YARN-2312-branch-2.8.patch, YARN-2312-wip.patch,
YARN-2312.1.patch, YARN-2312.2-2.patch, YARN-2312.2-3.patch,
YARN-2312.2.patch, YARN-2312.4.patch, YARN-2312.5.patch, YARN-2312.6.patch,
YARN-2312.7.patch
{{ContainerId#getId}} will only return partial value of containerId, only
sequence number of container id without epoch, after YARN-2229. We should
mark {{ContainerId#getId}} as deprecated and use
{{ContainerId#getContainerId}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
[
https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173208#comment-14173208
]
Vinod Kumar Vavilapalli commented on YARN-2496:
---
+1, looks good! Checking this in, let's make sure we follow up on the remaining
items.
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2496+YARN-2500.patch, YARN-2496+YARN-2500.patch,
YARN-2496-20141009-1.patch, YARN-2496-20141014-1.patch,
YARN-2496-20141015-1.patch, YARN-2496-20141015-2.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-2500) Miscellaneous changes in ResourceManager to support labels
[ https://issues.apache.org/jira/browse/YARN-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved YARN-2500. --- Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Committed this to trunk, branch-2 and branch-2.6. Thanks Wangda! Miscellaneous changes in ResourceManager to support labels -- Key: YARN-2500 URL: https://issues.apache.org/jira/browse/YARN-2500 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.6.0 Attachments: YARN-2500-20141009-1.patch, YARN-2500-20141009-2.patch, YARN-2500-20141010-1.patch, YARN-2500-20141014-1.patch, YARN-2500-20141015-1.patch, YARN-2500.patch, YARN-2500.patch, YARN-2500.patch, YARN-2500.patch, YARN-2500.patch This patches contains changes in ResourceManager to support labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2496) Changes for capacity scheduler to support allocate resource respect labels
/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
Changes for capacity scheduler to support allocate resource respect labels
--
Key: YARN-2496
URL: https://issues.apache.org/jira/browse/YARN-2496
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Fix For: 2.6.0
Attachments: YARN-2496+YARN-2500.patch, YARN-2496+YARN-2500.patch,
YARN-2496-20141009-1.patch, YARN-2496-20141014-1.patch,
YARN-2496-20141015-1.patch, YARN-2496-20141015-2.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch,
YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
This JIRA Includes:
- Add/parse labels option to {{capacity-scheduler.xml}} similar to other
options of queue like capacity/maximum-capacity, etc.
- Include a default-label-expression option in queue config, if an app
doesn't specify label-expression, default-label-expression of queue will be
used.
- Check if labels can be accessed by the queue when submit an app with
labels-expression to queue or update ResourceRequest with label-expression
- Check labels on NM when trying to allocate ResourceRequest on the NM with
label-expression
- Respect labels when calculate headroom/user-limit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2504) Support get/add/remove/change labels in RM admin CLI
[ https://issues.apache.org/jira/browse/YARN-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2504: - Attachment: YARN-2504-20141015-1.patch Support get/add/remove/change labels in RM admin CLI - Key: YARN-2504 URL: https://issues.apache.org/jira/browse/YARN-2504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2504-20141015-1.patch, YARN-2504.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2685) Resource on each label not correct when multiple NMs in a same host and some has label some not
[ https://issues.apache.org/jira/browse/YARN-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173228#comment-14173228 ] Vinod Kumar Vavilapalli commented on YARN-2685: --- Looks good, +1. Checking this in. Resource on each label not correct when multiple NMs in a same host and some has label some not --- Key: YARN-2685 URL: https://issues.apache.org/jira/browse/YARN-2685 Project: Hadoop YARN Issue Type: Sub-task Components: api, client, resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2685-20141013.1.patch I noticed there's one issue, when we have multiple NMs running in a same host, (say NM1-4 running in host1). And we specify some of them has label and some not, the total resource on label is not correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2504) Support get/add/remove/change labels in RM admin CLI
[ https://issues.apache.org/jira/browse/YARN-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173233#comment-14173233 ] Wangda Tan commented on YARN-2504: -- [~vinodkv], bq. Let's update the APIs to be named inline with the last patch at YARN-2544 - addToClusterNodeLabels etc. bq. Similarly the CLI options bq. We had dropped the clear-labels option? Addressed bq. getNodeToLabels() doesn't need to be in the admin-protocol or the admin-CLI bq. Similarly getLabels() Offline chat with Vinod, we can address that in a separated patch bq. Change the usage strings of the CLIs to instead say Comma separated list of node-labels Addressed bq. Not sure why we need doNotRetry here. The ClientRMProxy is the right layer to do this. bq. On a related note, we shouldn't do this try RM, fall back to local-store. We need to have modes in RMAdminCLI which can be used to run the Added an option directlyAccessNodeLabelStore to let admin explicitly specify if it need access local-store directly or RM. [~sunilg], I'm not sure what you meant, did you mean we need move the security verification code to RMNodeLabelsManager? Thanks, Wangda Support get/add/remove/change labels in RM admin CLI - Key: YARN-2504 URL: https://issues.apache.org/jira/browse/YARN-2504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2504-20141015-1.patch, YARN-2504.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2685) Resource on each label not correct when multiple NMs in a same host and some has label some not
[ https://issues.apache.org/jira/browse/YARN-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173234#comment-14173234 ] Hudson commented on YARN-2685: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6270 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6270/]) YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan. (vinodkv: rev b3056c266a628a65cf7ceb61b55ab3bd0a09baf2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java Resource on each label not correct when multiple NMs in a same host and some has label some not --- Key: YARN-2685 URL: https://issues.apache.org/jira/browse/YARN-2685 Project: Hadoop YARN Issue Type: Sub-task Components: api, client, resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.6.0 Attachments: YARN-2685-20141013.1.patch I noticed there's one issue, when we have multiple NMs running in a same host, (say NM1-4 running in host1). And we specify some of them has label and some not, the total resource on label is not correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2504) Support get/add/remove/change labels in RM admin CLI
[
https://issues.apache.org/jira/browse/YARN-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173236#comment-14173236
]
Sunil G commented on YARN-2504:
---
Hi [~gp.leftnoteasy]
{code}
AddToClusterNodeLabelsResponse response =
recordFactory.newRecordInstance(AddToClusterNodeLabelsResponse.class);
try {
rmContext.getNodeLabelManager().addToCluserNodeLabels(request.getNodeLabels());
RMAuditLogger
.logSuccess(user.getShortUserName(), argName, AdminService);
return response;
}
{code}
Here *response* object is created. Now labels which u received from incoming
request is passed to LabelManager and it may undergo a validation there. If any
problem of invalid label arises, how response object is populated. Here I was
not able to get that part of code.
Support get/add/remove/change labels in RM admin CLI
-
Key: YARN-2504
URL: https://issues.apache.org/jira/browse/YARN-2504
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2504-20141015-1.patch, YARN-2504.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2698) Move getClusterNodeLabels and getNodeToLabels to YARN CLI instead of RMAdminCLI
Wangda Tan created YARN-2698: Summary: Move getClusterNodeLabels and getNodeToLabels to YARN CLI instead of RMAdminCLI Key: YARN-2698 URL: https://issues.apache.org/jira/browse/YARN-2698 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan YARN RMAdminCLI and AdminService should have write API only, for other read APIs, they should be located at YARNCLI and RMClientService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2502) Changes in distributed shell to support specify labels
[
https://issues.apache.org/jira/browse/YARN-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173254#comment-14173254
]
Hadoop QA commented on YARN-2502:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674619/YARN-2502-20141013.1.patch
against trunk revision b3056c2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5416//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5416//console
This message is automatically generated.
Changes in distributed shell to support specify labels
--
Key: YARN-2502
URL: https://issues.apache.org/jira/browse/YARN-2502
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2502-20141009.1.patch, YARN-2502-20141009.2.patch,
YARN-2502-20141013.1.patch, YARN-2502.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2579) Both RM's state is Active , but 1 RM is not really active.
[ https://issues.apache.org/jira/browse/YARN-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173273#comment-14173273 ] Rohith commented on YARN-2579: -- Hi [~vinodkv], [~kasha], [~jianhe] Can this issue fix goes into release 2.6.0 please? If it appears,then one of the RM will be in hanged state. It is kind of deadlock between 2 threads where jstack does not show as deadlock. Both RM's state is Active , but 1 RM is not really active. -- Key: YARN-2579 URL: https://issues.apache.org/jira/browse/YARN-2579 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.5.1 Reporter: Rohith Assignee: Rohith Attachments: YARN-2579.patch, YARN-2579.patch I encountered a situaltion where both RM's web page was able to access and its state displayed as Active. But One of the RM's ActiveServices were stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2690) Make ReservationSystem and its dependent classes independent of Scheduler type
[ https://issues.apache.org/jira/browse/YARN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173279#comment-14173279 ] Subru Krishnan commented on YARN-2690: -- Thanks [~adhoot] for the patch. This is a good improvement as it minimizes the scheduler specific implementation for the reservation system. The patch overall looks good. Just a minor comment - can you move _initializePlan()_, _getReplanner()_, _getAgent()_ _getAdmissionPolicy()_ from _CapacityReservationSystem_ to _AbstractReservationSystem_ as now they are scheduler implementation agnostic. Make ReservationSystem and its dependent classes independent of Scheduler type Key: YARN-2690 URL: https://issues.apache.org/jira/browse/YARN-2690 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2690.001.patch A lot of common reservation classes depend on CapacityScheduler and specifically its configuration. This jira is to make them ready for other Schedulers by abstracting out the configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2100) Refactor the Timeline Server code for Kerberos + DT authentication
[ https://issues.apache.org/jira/browse/YARN-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved YARN-2100. --- Resolution: Duplicate YARN-2676 is going to reuse the common DT auth filter. Close this Jira as the duplicate of it. Refactor the Timeline Server code for Kerberos + DT authentication -- Key: YARN-2100 URL: https://issues.apache.org/jira/browse/YARN-2100 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The customized Kerberos + DT authentication of the timeline server largely refers to that of Http FS, therefore, there're a portion of duplicate code. We should think about refactor the code if it is necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over
[ https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1879: - Attachment: YARN-1879.25.patch Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over Key: YARN-1879 URL: https://issues.apache.org/jira/browse/YARN-1879 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jian He Assignee: Tsuyoshi OZAWA Priority: Critical Attachments: YARN-1879.1.patch, YARN-1879.1.patch, YARN-1879.11.patch, YARN-1879.12.patch, YARN-1879.13.patch, YARN-1879.14.patch, YARN-1879.15.patch, YARN-1879.16.patch, YARN-1879.17.patch, YARN-1879.18.patch, YARN-1879.19.patch, YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.20.patch, YARN-1879.21.patch, YARN-1879.22.patch, YARN-1879.23.patch, YARN-1879.23.patch, YARN-1879.24.patch, YARN-1879.25.patch, YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch, YARN-1879.7.patch, YARN-1879.8.patch, YARN-1879.9.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over
[
https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173342#comment-14173342
]
Hadoop QA commented on YARN-1879:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675219/YARN-1879.25.patch
against trunk revision 2894433.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5417//console
This message is automatically generated.
Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM
fail over
Key: YARN-1879
URL: https://issues.apache.org/jira/browse/YARN-1879
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Tsuyoshi OZAWA
Priority: Critical
Attachments: YARN-1879.1.patch, YARN-1879.1.patch,
YARN-1879.11.patch, YARN-1879.12.patch, YARN-1879.13.patch,
YARN-1879.14.patch, YARN-1879.15.patch, YARN-1879.16.patch,
YARN-1879.17.patch, YARN-1879.18.patch, YARN-1879.19.patch,
YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.20.patch,
YARN-1879.21.patch, YARN-1879.22.patch, YARN-1879.23.patch,
YARN-1879.23.patch, YARN-1879.24.patch, YARN-1879.25.patch,
YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch,
YARN-1879.7.patch, YARN-1879.8.patch, YARN-1879.9.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over
[
https://issues.apache.org/jira/browse/YARN-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173346#comment-14173346
]
Tsuyoshi OZAWA commented on YARN-1879:
--
Talked with Jian offline.
{quote}
In this case, token is expired for the application after finishing AM's
container and I think we don't need to handle it.
{quote}
I'd like to confirm whether finishApplicationMaster() can be issued after AM
containers exit. There are no such case, but finishApplicationMaster() can be
issued after RM's removing AM's entry in a following case:
1. RM1 saves the app in RMStateStore and then crashes.
2. FinishApplicationMasterResponse#isRegistered still return false.
3. The AM still retries the 2nd RM.
Thanks very much for clarifying, Jian. Attached a updated patch which includes
a test for retried finishApplicationMaster and a test for retried
registerApplicationMaster before and after RM-restart.
Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM
fail over
Key: YARN-1879
URL: https://issues.apache.org/jira/browse/YARN-1879
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Tsuyoshi OZAWA
Priority: Critical
Attachments: YARN-1879.1.patch, YARN-1879.1.patch,
YARN-1879.11.patch, YARN-1879.12.patch, YARN-1879.13.patch,
YARN-1879.14.patch, YARN-1879.15.patch, YARN-1879.16.patch,
YARN-1879.17.patch, YARN-1879.18.patch, YARN-1879.19.patch,
YARN-1879.2-wip.patch, YARN-1879.2.patch, YARN-1879.20.patch,
YARN-1879.21.patch, YARN-1879.22.patch, YARN-1879.23.patch,
YARN-1879.23.patch, YARN-1879.24.patch, YARN-1879.25.patch,
YARN-1879.3.patch, YARN-1879.4.patch, YARN-1879.5.patch, YARN-1879.6.patch,
YARN-1879.7.patch, YARN-1879.8.patch, YARN-1879.9.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
