[jira] [Created] (YARN-6091) the AppMaster register failed when use Docker on LinuxContainer
zhengchenyu created YARN-6091: - Summary: the AppMaster register failed when use Docker on LinuxContainer Key: YARN-6091 URL: https://issues.apache.org/jira/browse/YARN-6091 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, yarn Affects Versions: 2.8.0 Environment: CentOS Reporter: zhengchenyu Priority: Critical Fix For: 2.8.0 In some servers, When I use Docker on LinuxContainer, I found the aciton that AppMaster register to Resourcemanager failed. But didn't happen in other servers. I found the pclose (in container-executor.c) return different value in different server, even though the process which is launched by popen is running normally. Some server return 0, and others return 13. Because yarn regard the application as failed application when pclose return nonzero, and yarn will remove the AMRMToken, then the AppMaster register failed because Resourcemanager have removed this applicaiton's token. In container-executor.c, the judgement condition is whether the return code is zero. But man the pclose, the document tells that "pclose return -1" represent wrong. So I change the judgement condition, then slove this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6007) Unit test failures in Yarn due to java.lang.NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sonia Garudi resolved YARN-6007. Resolution: Fixed > Unit test failures in Yarn due to java.lang.NoClassDefFoundError > > > Key: YARN-6007 > URL: https://issues.apache.org/jira/browse/YARN-6007 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0-alpha2 > Environment: $ uname -a > Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 > 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Sonia Garudi > Labels: ppc64 > Attachments: YARN-6007.patch > > > In Hadoop Yarn project, 21 unit test failures are seen with below error : > java.lang.NoClassDefFoundError: Could not initialize class > org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer > java.lang.NullPointerException: null > at org.apache.hadoop.yarn.server.timelineservice.storage. > TestHBaseTimelineStorageApps.tearDownAfterClass(TestHBaseTimelineStorageApps.java:1847) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6092) Refreshing CapacityScheduler page throws NPE
Rohith Sharma K S created YARN-6092:
---
Summary: Refreshing CapacityScheduler page throws NPE
Key: YARN-6092
URL: https://issues.apache.org/jira/browse/YARN-6092
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
It is observe that RM CapacityScheduler page throw NPE in RM logs.
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$HealthBlock.render(CapacitySchedulerPage.java:551)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet._(Hamlet.java:30354)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(CapacitySchedulerPage.java:471)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh resolved YARN-5646. --- Resolution: Fixed Committed addendum patch to trunk and branch-2 > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch, YARN-5646.addendum.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/ [Jan 12, 2017 3:37:49 PM] (Arun Suresh) YARN-5646. Addendum patch to include missing changes to site.xml. -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestRollingUpgrade hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Timed out junit tests : org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [152K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [12K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/285/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6093) Invalid AMRM token exception when RM renew AMRMtoken and FederationRMFailoverProxyProvider failover
Botong Huang created YARN-6093: -- Summary: Invalid AMRM token exception when RM renew AMRMtoken and FederationRMFailoverProxyProvider failover Key: YARN-6093 URL: https://issues.apache.org/jira/browse/YARN-6093 Project: Hadoop YARN Issue Type: Bug Components: federation Reporter: Botong Huang Assignee: Botong Huang Priority: Minor Fix For: YARN-2915 AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid AMRMToken" exception. The bug is triggered when both conditions are met: 1. RM rolls master key and renews AMRMToken for a running AM. 2. Existing RPC connection between AMRMProxy and RM drops and attempt to reconnect via failover in FederationRMFailoverProxyProvider. Here's what happened: In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the initial AMRMToken issued by RM, and used it for initiating rmClient. Then we arrive at FederationRMFailoverProxyProvider.init(), a full copy of ugi tokens are saved locally, create an actual RM proxy and setup the RPC connection. Later when RM rolls master key and issues a new AMRMToken, DefaultRequestInterceptor.updateAMRMToken() updates it into the proxy ugi. However the new token is never used until the existing RPC connection between AMRMProxy and RM drops for other reasons (say master RM crashes). At this point, since the service name of the new AMRMToken is not yet set correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found no valid AMRMToken when trying to setup a new connection. We first hit a "Client cannot authenticate via:[TOKEN]" exception. This is expected. Next, FederationRMFailoverProxyProvider fails over, we reset the service token via ClientRMProxy.getRMAddress() and reconnect. Supposedly this would have worked. However since DefaultRequestInterceptor does not use the proxy user for later calls to rmClient, when performing failover in FederationRMFailoverProxyProvider, we are not in the proxy user. Currently the code solve the problem by reloading the current ugi with all tokens saved locally in originalTokens in method addOriginalTokens(). The problem is that the original AMRMToken loaded is no longer accepted by RM, and thus we keep hitting the "Invalid AMRMToken" exception until AM fails. The correct way is that rather than saving the original tokens in the proxy ugi, we save the original ugi itself. Every time we perform failover and create the new RM proxy, we use the original ugi, which is always loaded with the up-to-date AMRMToken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Not be able to receive notification mail for watch JIRA (YARN)
Hi yarn-devs, I'm not be able to receive any watched JIRA updates after JIRA maintenance completed, does anybody see the same issue? Thanks, Wangda
[jira] [Created] (YARN-6094) Update the coprocessor to be a dynamically loaded one
Vrushali C created YARN-6094: Summary: Update the coprocessor to be a dynamically loaded one Key: YARN-6094 URL: https://issues.apache.org/jira/browse/YARN-6094 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vrushali C Assignee: Vrushali C The timeline service v2 code base on yarn now uses hbase 1.2.4 after YARN-5976. With this version of hbase, system classes (starting with org.apache.hadoop) can be loaded as table level coprocessors. Hence we should update the timeline service coprocessor to be a dynamically loaded one instead of static loading. This involves code changes as well as documentation updates for deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6095) create a REST API that returns the clusters for a given app id
Sangjin Lee created YARN-6095: - Summary: create a REST API that returns the clusters for a given app id Key: YARN-6095 URL: https://issues.apache.org/jira/browse/YARN-6095 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee It would be good to have a timeline service REST endpoint that can return the list of clusters for a given app id. This becomes possible after YARN-5378 is in. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6096) getCpuFrequency for windows returning incorrect value
Roni Burd created YARN-6096: --- Summary: getCpuFrequency for windows returning incorrect value Key: YARN-6096 URL: https://issues.apache.org/jira/browse/YARN-6096 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Environment: windows Reporter: Roni Burd Priority: Minor Fix For: 3.0.0-alpha1 getCpuFrequency in WindowsResourceCalculatorPlugin returns -1 instead of the actual parsed value -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6097) Add support for directories in the Shared Cache
Chris Trezzo created YARN-6097: -- Summary: Add support for directories in the Shared Cache Key: YARN-6097 URL: https://issues.apache.org/jira/browse/YARN-6097 Project: Hadoop YARN Issue Type: Bug Reporter: Chris Trezzo Add support for directories in the shared cache. If a LocalResource URL points to a directory, the directory structure is preserved during localization on the node manager. Currently, the shared cache does not support directories and will fail to upload the URL to the cache if shouldBeUploadedToSharedCache is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6098) Add documentation for Delete Queue
Naganarasimha G R created YARN-6098: --- Summary: Add documentation for Delete Queue Key: YARN-6098 URL: https://issues.apache.org/jira/browse/YARN-6098 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler, documentation Reporter: Naganarasimha G R Assignee: Naganarasimha G R As per the discussion in YARN-5556, we need to document steps for deleting a queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
RE: Not be able to receive notification mail for watch JIRA (YARN)
Yes, Even I noticed same. Can we raise a ticket in INFRA..? Marking common-dev also in loop. Thanks and Regards Brahma Reddy Battula -Original Message- From: Wangda Tan [mailto:wheele...@gmail.com] Sent: 14 January 2017 02:15 To: yarn-dev@hadoop.apache.org Subject: Not be able to receive notification mail for watch JIRA (YARN) Hi yarn-devs, I'm not be able to receive any watched JIRA updates after JIRA maintenance completed, does anybody see the same issue? Thanks, Wangda - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
