[jira] [Created] (YARN-2955) mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector
Rocju created YARN-2955: --- Summary: mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector Key: YARN-2955 URL: https://issues.apache.org/jira/browse/YARN-2955 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Environment: cdh5.1.0 Reporter: Rocju 2014-12-12 02:26:55,047 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector@dcnn2:23188 2014-12-12 02:26:55,052 WARN mortbay.log (Slf4jLog.java:warn(89)) - EXCEPTION java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279) at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154) at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904) at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:755) at java.io.Writer.write(Writer.java:157) at java.io.PrintWriter.newLine(PrintWriter.java:480) at java.io.PrintWriter.println(PrintWriter.java:629) at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._p(HamletImpl.java:110) at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$SCRIPT._(Hamlet.java:454) at org.apache.hadoop.yarn.server.resourcemanager.webapp.AppsBlock.render(AppsBlock.java:119) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:66) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:76) at org.apache.hadoop.yarn.webapp.View.render(View.java:235) at org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:40) at org.apache.hadoop.yarn.webapp.hamlet.Hamlet._(Hamlet.java:30347) at org.apache.hadoop.yarn.server.resourcemanager.webapp.AppsBlockWithMetrics.render(AppsBlockWithMetrics.java:29) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:66) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:76) at org.apache.hadoop.yarn.webapp.View.render(View.java:235) at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117) at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845) at org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56) at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82) at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:197) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:156) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMDispatcher.service(RMDispatcher.java:77) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
[jira] [Updated] (YARN-2738) Add FairReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated YARN-2738: Attachment: YARN-2738.004.patch Addressed all feedback Allowed configuring the class names at the global level and only ability to turn on/off reservability at queue level. Disallows reservable and dynamic queues at the same time Additional unit tests verifying the same Add FairReservationSystem for FairScheduler --- Key: YARN-2738 URL: https://issues.apache.org/jira/browse/YARN-2738 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2738.001.patch, YARN-2738.002.patch, YARN-2738.003.patch, YARN-2738.004.patch Need to create a FairReservationSystem that will implement ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2738) Add FairReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243871#comment-14243871 ] Anubhav Dhoot commented on YARN-2738: - resolveReservationQueueName and handleMoveToPlanQueue will be addressed in the YARN-2881 Add FairReservationSystem for FairScheduler --- Key: YARN-2738 URL: https://issues.apache.org/jira/browse/YARN-2738 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2738.001.patch, YARN-2738.002.patch, YARN-2738.003.patch, YARN-2738.004.patch Need to create a FairReservationSystem that will implement ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2738) Add FairReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243931#comment-14243931 ] Hadoop QA commented on YARN-2738: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686809/YARN-2738.004.patch against trunk revision bda748a. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 15 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6102//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6102//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6102//console This message is automatically generated. Add FairReservationSystem for FairScheduler --- Key: YARN-2738 URL: https://issues.apache.org/jira/browse/YARN-2738 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2738.001.patch, YARN-2738.002.patch, YARN-2738.003.patch, YARN-2738.004.patch Need to create a FairReservationSystem that will implement ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2356: -- Attachment: 0004-YARN-2356.patch Hi [~devaraj.k] Thank you for the update. I have fixed the warnings yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at $Proxy12.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:428) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:76) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException): Application with id 'application_1402668848165_0015' doesn't exist in RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[ https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243968#comment-14243968 ] Hudson commented on YARN-2917: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/]) YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * hadoop-yarn-project/CHANGES.txt Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Key: YARN-2917 URL: https://issues.apache.org/jira/browse/YARN-2917 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith Priority: Critical Fix For: 2.7.0 Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch I encoutered scenario where RM hanged while shutting down and keep on logging {{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[ https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243964#comment-14243964 ] Hudson commented on YARN-2243: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/]) YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in (devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor Key: YARN-2243 URL: https://issues.apache.org/jira/browse/YARN-2243 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.1 Reporter: Ted Yu Assignee: Devaraj K Priority: Minor Attachments: YARN-2243.patch, YARN-2243.patch {code} public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, String user, Queue queue, ActiveUsersManager activeUsersManager, RMContext rmContext) { Preconditions.checkNotNull(RMContext should not be null, rmContext); {code} Order of arguments is wrong for Preconditions.checkNotNull(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[ https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244024#comment-14244024 ] Hudson commented on YARN-2917: -- FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/773/]) YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Key: YARN-2917 URL: https://issues.apache.org/jira/browse/YARN-2917 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith Priority: Critical Fix For: 2.7.0 Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch I encoutered scenario where RM hanged while shutting down and keep on logging {{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[ https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244020#comment-14244020 ] Hudson commented on YARN-2243: -- FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/773/]) YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in (devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/CHANGES.txt Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor Key: YARN-2243 URL: https://issues.apache.org/jira/browse/YARN-2243 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.1 Reporter: Ted Yu Assignee: Devaraj K Priority: Minor Attachments: YARN-2243.patch, YARN-2243.patch {code} public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, String user, Queue queue, ActiveUsersManager activeUsersManager, RMContext rmContext) { Preconditions.checkNotNull(RMContext should not be null, rmContext); {code} Order of arguments is wrong for Preconditions.checkNotNull(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) Deadlock in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244050#comment-14244050 ] Rohith commented on YARN-2946: -- Another deadlock detected in different flow after fixing previous deadlock :-( Basically, by convention all the locks should acquire from *StateMachine.doTransition() - zkRMStateStore.class* or directly zkRMStateStore.class , but in {{RMStateStore#isFencedState()}} method locking order is reversed i.e *zkRMStateStore.class - StateMachine.doTransition()* {noformat} Found one Java-level deadlock: = org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread: waiting to lock monitor 0x00e55698 (object 0xc0272cb0, a org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine), which is held by AsyncDispatcher event handler AsyncDispatcher event handler: waiting to lock monitor 0x013adcf8 (object 0xc0272b10, a org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore), which is held by org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread Java stack information for the threads listed above: === org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread: at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.getCurrentState(StateMachineFactory.java:442) - waiting to lock 0xc0272cb0 (a org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.isFencedState(RMStateStore.java:693) - locked 0xc0272b10 (a org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread.run(ZKRMStateStore.java:1020) AsyncDispatcher event handler: at java.lang.Object.wait(Native Method) - waiting on 0xc0272b10 (a org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1043) - locked 0xc0272b10 (a org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1070) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:906) - locked 0xc0272b10 (a org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:920) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:929) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.storeApplicationStateInternal(ZKRMStateStore.java:608) - locked 0xc0272b10 (a org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:146) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:131) at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) - locked 0xc0272cb0 (a org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:699) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:754) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:749) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock. {noformat} Deadlock in ZKRMStateStore -- Key: YARN-2946 URL:
[jira] [Created] (YARN-2956) Some yarn-site index linked pages are difficult to discover because are not in the side bar
Remus Rusanu created YARN-2956: -- Summary: Some yarn-site index linked pages are difficult to discover because are not in the side bar Key: YARN-2956 URL: https://issues.apache.org/jira/browse/YARN-2956 Project: Hadoop YARN Issue Type: Bug Components: documentation Affects Versions: 2.6.0 Reporter: Remus Rusanu Priority: Minor The yarn-site index.apt.vm page is difficult to 'stumble upon' because the hadoop.apache.org/ sidebar navigation does not link to it. One needs to know the URL http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ to land on it. The links from the index page do not match the links from the side bar, so some pages are quickly accessible (from sidebar). I propose the links from the index.apt.vm to match the links from the YARN side bar subsection (Ideally through one single definition file, but I don't understand the APT generation process well enough to call out how this can be achieved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) Deadlock in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244098#comment-14244098 ] Rohith commented on YARN-2946: -- Above 2 deadlock's can be directly fixed by removing *synchronized* keyword which was not really required. But I see there many other potential deadlocks which can appear easily in the following methods. Below all methods does reverse locking i.e *zkRMStateStore.class - StateMachine.doTransition()* through {{RMStateStore#isFencedState()}} # Method {{RMStateStore#storeRMDTMasterKey()}} # Method {{RMStateStore#removeRMDTMasterKey()}} # Method {{RMStateStore#storeRMDelegationTokenAndSequenceNumber()}} # Method {{RMStateStore#removeRMDelegationToken()}} # Method {{RMStateStore#updateRMDelegationTokenAndSequenceNumber()}} # Method {{ZKRMStateStore#storeOrUpdateAMRMTokenSecretManagerState()}} So only fixing 1 or 2 deadlock flows does not really fix other potential dead lock issues. *I propose following solution to handle all these deadlock flows* Option-1 : # For all above mentioned method's causing deadlock , introduce StateMachine in RMStateStore like handling application store. So all the execution flows from StateMachine-zkRMStateStore.class. # Along with 1st , StateMachine should be guarded with Read-Write lock. Option-2 : # Fix the visible eadlocks i.e 2 found in this jira. And Option-1 do in separate improvement task. Handling all the deadlock flows, i would like to do in one umbrella jira. This is to ensure we do not miss any these deadlock flows. Please let me your suggestions/thoughts? Deadlock in ZKRMStateStore -- Key: YARN-2946 URL: https://issues.apache.org/jira/browse/YARN-2946 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Reporter: Rohith Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch, TestYARN2946.java Found one deadlock in ZKRMStateStore. # Initial stage zkClient is null because of zk disconnected event. # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to re establish zookeeper connection either via synconnected or expired event, it is highly possible that any other thred can obtain lock on {{ZKRMStateStore.this}} from state machine transition events. This cause Deadlock in ZKRMStateStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2946) DeadLock's in RMStateStore-ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated YARN-2946: - Summary: DeadLock's in RMStateStore-ZKRMStateStore (was: Deadlock in ZKRMStateStore) DeadLock's in RMStateStore-ZKRMStateStore --- Key: YARN-2946 URL: https://issues.apache.org/jira/browse/YARN-2946 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Reporter: Rohith Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch, TestYARN2946.java Found one deadlock in ZKRMStateStore. # Initial stage zkClient is null because of zk disconnected event. # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to re establish zookeeper connection either via synconnected or expired event, it is highly possible that any other thred can obtain lock on {{ZKRMStateStore.this}} from state machine transition events. This cause Deadlock in ZKRMStateStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLock's in RMStateStore-ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244142#comment-14244142 ] Varun Saxena commented on YARN-2946: [~rohithsharma], good catch. As you said {{synchronized}} can be removed from {{RMStateStore#isFencedState()}}. Regarding the methods you listed out, I think we can avoid reverse locking i.e. *ZKRMStateStore.class - StateMachine.doTransition()* by making the following changes. # Remove {{synchronized}} keyword from each one of the methods listed above. # Separate out State machine synchronization(invoked by call to {{isFencedState}} {{notifyStoreOperationFailed}}) and ZKRMStateStore synchronization by putting the relevant code in a synchronized block. For example, code for RMStateStore#storeRMDTMasterKey() can be changed as under : {code:title=RMStateStore.java|borderStyle=solid} public void storeRMDTMasterKey(DelegationKey delegationKey) { if(isFencedState()) { LOG.info(State store is in Fenced state. Can't store RM Delegation + Token Master key.); return; } try { synchronized(this) { storeRMDTMasterKeyState(delegationKey); } } catch (Exception e) { notifyStoreOperationFailed(e); } } {code} DeadLock's in RMStateStore-ZKRMStateStore --- Key: YARN-2946 URL: https://issues.apache.org/jira/browse/YARN-2946 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Reporter: Rohith Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch, TestYARN2946.java Found one deadlock in ZKRMStateStore. # Initial stage zkClient is null because of zk disconnected event. # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to re establish zookeeper connection either via synconnected or expired event, it is highly possible that any other thred can obtain lock on {{ZKRMStateStore.this}} from state machine transition events. This cause Deadlock in ZKRMStateStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[ https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244176#comment-14244176 ] Falk Scheerschmidt commented on YARN-1842: -- Same problem here with Hadoop 2.6 and Zookeeper 3.4 on Debian. InvalidApplicationMasterRequestException raised during AM-requested shutdown Key: YARN-1842 URL: https://issues.apache.org/jira/browse/YARN-1842 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Steve Loughran Priority: Minor Attachments: hoyalogs.tar.gz Report of the RM raising a stack trace [https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM could just swallow this and exit, but it could be a sign of a race condition YARN-side, or maybe just in the RM client code/AM dual signalling the shutdown. I haven't replicated this myself; maybe the stack will help track down the problem. Otherwise: what is the policy YARN apps should adopt for AM's handling errors on shutdown? go straight to an exit(-1)? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[ https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244190#comment-14244190 ] Hudson commented on YARN-2243: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/]) YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in (devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/CHANGES.txt Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor Key: YARN-2243 URL: https://issues.apache.org/jira/browse/YARN-2243 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.1 Reporter: Ted Yu Assignee: Devaraj K Priority: Minor Attachments: YARN-2243.patch, YARN-2243.patch {code} public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, String user, Queue queue, ActiveUsersManager activeUsersManager, RMContext rmContext) { Preconditions.checkNotNull(RMContext should not be null, rmContext); {code} Order of arguments is wrong for Preconditions.checkNotNull(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[ https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244199#comment-14244199 ] Hudson commented on YARN-2243: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/]) YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in (devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor Key: YARN-2243 URL: https://issues.apache.org/jira/browse/YARN-2243 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.1 Reporter: Ted Yu Assignee: Devaraj K Priority: Minor Attachments: YARN-2243.patch, YARN-2243.patch {code} public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, String user, Queue queue, ActiveUsersManager activeUsersManager, RMContext rmContext) { Preconditions.checkNotNull(RMContext should not be null, rmContext); {code} Order of arguments is wrong for Preconditions.checkNotNull(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[ https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244203#comment-14244203 ] Hudson commented on YARN-2917: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/]) YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * hadoop-yarn-project/CHANGES.txt Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Key: YARN-2917 URL: https://issues.apache.org/jira/browse/YARN-2917 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith Priority: Critical Fix For: 2.7.0 Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch I encoutered scenario where RM hanged while shutting down and keep on logging {{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[ https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244194#comment-14244194 ] Hudson commented on YARN-2917: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/]) YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Key: YARN-2917 URL: https://issues.apache.org/jira/browse/YARN-2917 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith Priority: Critical Fix For: 2.7.0 Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch I encoutered scenario where RM hanged while shutting down and keep on logging {{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLock's in RMStateStore-ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244225#comment-14244225 ] Varun Saxena commented on YARN-2946: Frankly, as all of these methods *merely have a single line calling another method* in {{ZKRMStateStore}}, in addition to call to isFencedState and notifyStoreOperationFailed. Hence, we *do not even need a synchronized block* in these methods in RMStateStore. Just make the relevant method in ZKRMStateStore *synchronized*. DeadLock's in RMStateStore-ZKRMStateStore --- Key: YARN-2946 URL: https://issues.apache.org/jira/browse/YARN-2946 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Reporter: Rohith Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch, TestYARN2946.java Found one deadlock in ZKRMStateStore. # Initial stage zkClient is null because of zk disconnected event. # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to re establish zookeeper connection either via synconnected or expired event, it is highly possible that any other thred can obtain lock on {{ZKRMStateStore.this}} from state machine transition events. This cause Deadlock in ZKRMStateStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[ https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244246#comment-14244246 ] Hudson commented on YARN-2917: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/]) YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * hadoop-yarn-project/CHANGES.txt Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Key: YARN-2917 URL: https://issues.apache.org/jira/browse/YARN-2917 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith Priority: Critical Fix For: 2.7.0 Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch I encoutered scenario where RM hanged while shutting down and keep on logging {{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[ https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244242#comment-14244242 ] Hudson commented on YARN-2243: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/]) YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in (devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/CHANGES.txt Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor Key: YARN-2243 URL: https://issues.apache.org/jira/browse/YARN-2243 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.1 Reporter: Ted Yu Assignee: Devaraj K Priority: Minor Attachments: YARN-2243.patch, YARN-2243.patch {code} public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, String user, Queue queue, ActiveUsersManager activeUsersManager, RMContext rmContext) { Preconditions.checkNotNull(RMContext should not be null, rmContext); {code} Order of arguments is wrong for Preconditions.checkNotNull(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[ https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244269#comment-14244269 ] Hudson commented on YARN-2243: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/]) YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in (devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/CHANGES.txt Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor Key: YARN-2243 URL: https://issues.apache.org/jira/browse/YARN-2243 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.1 Reporter: Ted Yu Assignee: Devaraj K Priority: Minor Attachments: YARN-2243.patch, YARN-2243.patch {code} public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, String user, Queue queue, ActiveUsersManager activeUsersManager, RMContext rmContext) { Preconditions.checkNotNull(RMContext should not be null, rmContext); {code} Order of arguments is wrong for Preconditions.checkNotNull(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[ https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244273#comment-14244273 ] Hudson commented on YARN-2917: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/]) YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev 614b6afea450ebb897fbb2519c6f02e13b9bd12d) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook Key: YARN-2917 URL: https://issues.apache.org/jira/browse/YARN-2917 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith Priority: Critical Fix For: 2.7.0 Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch I encoutered scenario where RM hanged while shutting down and keep on logging {{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain.}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2952) Incorrect version check in RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1427#comment-1427 ] Rohith commented on YARN-2952: -- +1 for issue, always base major version is considered as 1. But in real use cases, base major version can be greater than 2 or 3 which installation itself will fail. Incorrect version check in RMStateStore --- Key: YARN-2952 URL: https://issues.apache.org/jira/browse/YARN-2952 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Rohith In RMStateStore#checkVersion: if we modify tCURRENT_VERSION_INFO to 2.0, it'll still store the version as 1.0 which is incorrect; The same thing might happen to NM store, timeline store. {code} // if there is no version info, treat it as 1.0; if (loadedVersion == null) { loadedVersion = Version.newInstance(1, 0); } if (loadedVersion.isCompatibleTo(getCurrentVersion())) { LOG.info(Storing RM state version info + getCurrentVersion()); storeVersion(); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2912) Jersey Tests failing with port in use
[ https://issues.apache.org/jira/browse/YARN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244470#comment-14244470 ] Hudson commented on YARN-2912: -- FAILURE: Integrated in Hadoop-trunk-Commit #6706 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6706/]) YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) (stevel: rev 3681de203949f84c9fa4a6df49948dbf6980c9ba) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/JerseyTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java * hadoop-yarn-project/CHANGES.txt Jersey Tests failing with port in use - Key: YARN-2912 URL: https://issues.apache.org/jira/browse/YARN-2912 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.0.0 Environment: jenkins on java 8 Reporter: Steve Loughran Assignee: Varun Saxena Fix For: 2.7.0 Attachments: YARN-2912.patch Jersey tests like TestNMWebServices apps are failing with port in use. The jersey test runner appears to always use the same port unless a system property is set to point to a different one. Every test should really be changing that sysprop in a @Before method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2952) Incorrect version check in RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244491#comment-14244491 ] Zhijie Shen commented on YARN-2952: --- This problematic code has be propagated to multiple places where leveldb is used, which need to be fixed together. Incorrect version check in RMStateStore --- Key: YARN-2952 URL: https://issues.apache.org/jira/browse/YARN-2952 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Rohith In RMStateStore#checkVersion: if we modify tCURRENT_VERSION_INFO to 2.0, it'll still store the version as 1.0 which is incorrect; The same thing might happen to NM store, timeline store. {code} // if there is no version info, treat it as 1.0; if (loadedVersion == null) { loadedVersion = Version.newInstance(1, 0); } if (loadedVersion.isCompatibleTo(getCurrentVersion())) { LOG.info(Storing RM state version info + getCurrentVersion()); storeVersion(); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2912) Jersey Tests failing with port in use
[ https://issues.apache.org/jira/browse/YARN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244551#comment-14244551 ] Hudson commented on YARN-2912: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #39 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/39/]) YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) (stevel: rev 3681de203949f84c9fa4a6df49948dbf6980c9ba) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/JerseyTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java Jersey Tests failing with port in use - Key: YARN-2912 URL: https://issues.apache.org/jira/browse/YARN-2912 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.0.0 Environment: jenkins on java 8 Reporter: Steve Loughran Assignee: Varun Saxena Fix For: 2.7.0 Attachments: YARN-2912.patch Jersey tests like TestNMWebServices apps are failing with port in use. The jersey test runner appears to always use the same port unless a system property is set to point to a different one. Every test should really be changing that sysprop in a @Before method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-2938: --- Attachment: FindBugs Report.html Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice -- Key: YARN-2938 URL: https://issues.apache.org/jira/browse/YARN-2938 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: FindBugs Report.html, YARN-2938.001.patch, YARN-2938.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244568#comment-14244568 ] Varun Saxena commented on YARN-2938: Report for RM attached. Inconsistent Synchronization I will verify once again. Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice -- Key: YARN-2938 URL: https://issues.apache.org/jira/browse/YARN-2938 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: FindBugs Report.html, YARN-2938.001.patch, YARN-2938.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244567#comment-14244567 ] Hadoop QA commented on YARN-2938: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686669/YARN-2938.002.patch against trunk revision 3681de2. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6103//console This message is automatically generated. Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice -- Key: YARN-2938 URL: https://issues.apache.org/jira/browse/YARN-2938 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: FindBugs Report.html, YARN-2938.001.patch, YARN-2938.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2850) DistributedShell: Add support for disk I/O request
[ https://issues.apache.org/jira/browse/YARN-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2850: -- Attachment: YARN-2850-1.patch Attach the first patch for review, based on YARN-2618. DistributedShell: Add support for disk I/O request -- Key: YARN-2850 URL: https://issues.apache.org/jira/browse/YARN-2850 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2850-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2851) YarnClient: Add support for disk I/O resource/request information
[ https://issues.apache.org/jira/browse/YARN-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2851: -- Attachment: YARN-2851-1.patch Put an initial patch for review, based on YARN-2618. YarnClient: Add support for disk I/O resource/request information - Key: YARN-2851 URL: https://issues.apache.org/jira/browse/YARN-2851 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2851-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2852) WebUI Metrics: Add disk I/O resource information to the web ui and metrics
[ https://issues.apache.org/jira/browse/YARN-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2852: -- Summary: WebUI Metrics: Add disk I/O resource information to the web ui and metrics (was: WebUI: Add disk I/O resource information to the web ui) WebUI Metrics: Add disk I/O resource information to the web ui and metrics Key: YARN-2852 URL: https://issues.apache.org/jira/browse/YARN-2852 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2852-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2852) WebUI Metrics: Add disk I/O resource information to the web ui and metrics
[ https://issues.apache.org/jira/browse/YARN-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2852: -- Attachment: YARN-2852-1.patch Patch for review, based on YARN-2618. WebUI Metrics: Add disk I/O resource information to the web ui and metrics Key: YARN-2852 URL: https://issues.apache.org/jira/browse/YARN-2852 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2852-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
Ray Chiang created YARN-2957: Summary: Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244628#comment-14244628 ] Zhijie Shen commented on YARN-2937: --- 1. What's the problem around Random? {code} -Random r = new Random(); -long randomPosition = Math.abs(r.nextLong()) % totalAvailable; +long randomPosition = Math.abs(RandomUtils.nextLong()) % totalAvailable; {code} 2. Can we use IOUtils? {code} - // Close the streams - try { -in.close(); - } catch (IOException e2) { -LOG.warn(Error closing the stream: + getMtabFileName(), e2); + if(in != null) { +// Close the streams +try { + in.close(); +} catch (IOException e2) { + LOG.warn(Error closing the stream: + getMtabFileName(), e2); +} {code} 3. Is this removed because bufReader will consequently close fileReader too? {code} - if (fileReader != null) { -fileReader.close(); - } {code} 4. Again, would you mind explaining a bit about what have been excluded in findbugs-exclude.xml? Thanks! Fix new findbugs warnings in hadoop-yarn-nodemanager Key: YARN-2937 URL: https://issues.apache.org/jira/browse/YARN-2937 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: HADOOP-11373.patch, YARN-2937.001.patch, YARN-2937.002.patch, YARN-2937.003.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[ https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated YARN-2950: -- Attachment: YARN-2950-1.patch Updating the error message as Harsh suggested. Change message to mandate, not suggest JS requirement on UI --- Key: YARN-2950 URL: https://issues.apache.org/jira/browse/YARN-2950 Project: Hadoop YARN Issue Type: Improvement Components: webapp Affects Versions: 2.5.0 Reporter: Harsh J Assignee: Dustin Cote Priority: Minor Labels: newbie Attachments: YARN-2950-1.patch Most of YARN's UIs do not work with JavaScript disabled on the browser, cause they appear to send back data as JS arrays instead of within the actual HTML content. The JQueryUI prints only a mild warning about this suggesting that {{This page works best with javascript enabled.}}, when in fact it ought to be {{This page will not function without javascript enabled. Please enable javascript on your browser.}} or something as such (more direct). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2620) FairScheduler: Add disk I/O resource to the DRF implementation
[ https://issues.apache.org/jira/browse/YARN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2620: -- Attachment: YARN-2620-1.patch FairScheduler: Add disk I/O resource to the DRF implementation -- Key: YARN-2620 URL: https://issues.apache.org/jira/browse/YARN-2620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2620-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2620) FairScheduler: Add disk I/O resource to the DRF implementation
[ https://issues.apache.org/jira/browse/YARN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244664#comment-14244664 ] Wei Yan commented on YARN-2620: --- Initial patch, based on YARN-2168. Some testcases fail, should be ok after we revisit the configuration in YARN-2941. FairScheduler: Add disk I/O resource to the DRF implementation -- Key: YARN-2620 URL: https://issues.apache.org/jira/browse/YARN-2620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2620-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2620) FairScheduler: Add disk I/O resource to the DRF implementation
[ https://issues.apache.org/jira/browse/YARN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244666#comment-14244666 ] Wei Yan commented on YARN-2620: --- typo... should be YARN-2618. FairScheduler: Add disk I/O resource to the DRF implementation -- Key: YARN-2620 URL: https://issues.apache.org/jira/browse/YARN-2620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2620-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-2957: - Attachment: HADOOP-11399.002.patch Cleaned up the javadoc. Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-2957: - Attachment: (was: HADOOP-11399.002.patch) Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-2957: - Attachment: YARN-2957.001.patch Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: YARN-2957.001.patch Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[ https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244712#comment-14244712 ] Hadoop QA commented on YARN-2950: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686924/YARN-2950-1.patch against trunk revision 3681de2. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 25 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6104//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6104//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6104//console This message is automatically generated. Change message to mandate, not suggest JS requirement on UI --- Key: YARN-2950 URL: https://issues.apache.org/jira/browse/YARN-2950 Project: Hadoop YARN Issue Type: Improvement Components: webapp Affects Versions: 2.5.0 Reporter: Harsh J Assignee: Dustin Cote Priority: Minor Labels: newbie Attachments: YARN-2950-1.patch Most of YARN's UIs do not work with JavaScript disabled on the browser, cause they appear to send back data as JS arrays instead of within the actual HTML content. The JQueryUI prints only a mild warning about this suggesting that {{This page works best with javascript enabled.}}, when in fact it ought to be {{This page will not function without javascript enabled. Please enable javascript on your browser.}} or something as such (more direct). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2944: --- Attachment: YARN-2944-trunk-v2.patch Attached is V2 to address [~sjlee0]'s comments and to fix unit tests. SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2944: --- Attachment: YARN-2944-trunk-v2.patch SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2944: --- Attachment: (was: YARN-2944-trunk-v2.patch) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244813#comment-14244813 ] Hadoop QA commented on YARN-2944: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686947/YARN-2944-trunk-v2.patch against trunk revision 46612c7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6105//console This message is automatically generated. SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244840#comment-14244840 ] Hadoop QA commented on YARN-2944: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686950/YARN-2944-trunk-v2.patch against trunk revision 46612c7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6106//console This message is automatically generated. SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2003) Support to process Job priority from Submission Context in AppAttemptAddedSchedulerEvent [RM side]
[ https://issues.apache.org/jira/browse/YARN-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244854#comment-14244854 ] Ashwin Shankar commented on YARN-2003: -- [~sunilg], Thanks for letting me know. Just wanted to emphasis that we are really interested in getting app priorities for fair scheduler committed soon, since we have few usecases in our production cluster that need this. Please let me know if you need help with the common code between the schedulers. Support to process Job priority from Submission Context in AppAttemptAddedSchedulerEvent [RM side] -- Key: YARN-2003 URL: https://issues.apache.org/jira/browse/YARN-2003 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G AppAttemptAddedSchedulerEvent should be able to receive the Job Priority from Submission Context and store. Later this can be used by Scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2944: --- Attachment: YARN-2944-trunk-v2.patch Note: The InMemorySCMStore(AppChecker appChecker) constructor actually needs to be public because it is used in TestSharedCacheUploaderService and TestClientSCMProtocolService. I marked it visible for testing. I did reduce the scope of the SCMStore constructor. SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2203: --- Attachment: SCMUI-trunk-v4.png Attaching v4 screen shot of the overview page. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2203: --- Attachment: YARN-2203-trunk-v4.patch Attaching v4 of the patch. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch, YARN-2203-trunk-v4.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244941#comment-14244941 ] Hadoop QA commented on YARN-2944: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686961/YARN-2944-trunk-v2.patch against trunk revision 46612c7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:red}-1 javac{color}. The applied patch generated 1217 javac compiler warnings (more than the trunk's current 155 warnings). {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 49 warning messages. See https://builds.apache.org/job/PreCommit-YARN-Build/6107//artifact/patchprocess/diffJavadocWarnings.txt for details. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 25 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6107//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6107//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6107//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6107//console This message is automatically generated. SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException:
[jira] [Commented] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244946#comment-14244946 ] Hadoop QA commented on YARN-2203: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686964/YARN-2203-trunk-v4.patch against trunk revision 7784b10. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:red}-1 eclipse:eclipse{color}. The patch failed to build with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6108//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6108//console This message is automatically generated. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch, YARN-2203-trunk-v4.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244953#comment-14244953 ] Chris Trezzo commented on YARN-2944: Findbugs/javac/javadoc warnings seem to be erroneous/unrelated. The patch should be good to go. SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244964#comment-14244964 ] Chris Trezzo commented on YARN-2203: eclipse:eclipse works fine locally. Also, this patch has been tested manually using a psudo-distributed cluster. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch, YARN-2203-trunk-v4.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[ https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244978#comment-14244978 ] Sangjin Lee commented on YARN-2944: --- LGTM. Thanks [~ctrezzo]! SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance -- Key: YARN-2944 URL: https://issues.apache.org/jira/browse/YARN-2944 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Minor Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create the SCMStore service. Unfortunately the SCMStore class does not have a 0-argument constructor. On startup, the SCM fails with the following: {noformat} 14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager failed in state INITED; cause: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more 14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting SharedCacheManager java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init() at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 4 more {noformat} This JIRA is to add a 0-argument constructor to SCMStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245019#comment-14245019 ] Hadoop QA commented on YARN-2356: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12686822/0004-YARN-2356.patch against trunk revision c78e3a7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 10 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6109//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6109//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6109//console This message is automatically generated. yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[jira] [Created] (YARN-2959) Fair Scheduler fifo option can violate FIFO behavior and cause deadlock among jobs
Ashwin Shankar created YARN-2959: Summary: Fair Scheduler fifo option can violate FIFO behavior and cause deadlock among jobs Key: YARN-2959 URL: https://issues.apache.org/jira/browse/YARN-2959 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Ashwin Shankar We have a cluster which run jobs in fifo order(due to the nature of those jobs) using Fair scheduler's fifo option. Recently we found jobs deadlocked in the cluster, here is what happened : There were two jobs,say A and B. A was submitted before B. Both were in PENDING state since the cluster was busy. When containers freed up, the two pending jobs got their AM containers at about the same time. However Job B's AM or appattempt1 registered with RM a little earlier than Job A and grabbed available containers at that time, and satisfied a fraction of its requirement. Note, JobB can't make progress until it gets all its requirement satisfied. Next, JobA's appattempt1 registered with RM and since JobA was submitted earlier, RM stops allocating containers to JobB and starts allocating to JobA, satisfying a fraction of its requirement as well. Now together jobA,jobB hold the entire cluster, but neither can progress and are deadlocked since their resource requests are partially satisfied. Note:Above is an example with 2 jobs, however the deadlock can happen with n jobs : J1..Jn if the sequence of AM registration is Jn, J(n-1),..J1. Solution : one proposed solution is to order the fifo queue by appattempt start/register time instead of app submit time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2958) RMStateStore seems to unnecessarily and wronly store sequence number separately
Zhijie Shen created YARN-2958: - Summary: RMStateStore seems to unnecessarily and wronly store sequence number separately Key: YARN-2958 URL: https://issues.apache.org/jira/browse/YARN-2958 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Zhijie Shen It seems that RMStateStore updates last sequence number when storing or updating each individual DT, to recover the latest sequence number when RM restarting. First, the current logic seems to be problematic: {code} public synchronized void updateRMDelegationTokenAndSequenceNumber( RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate, int latestSequenceNumber) { if(isFencedState()) { LOG.info(State store is in Fenced state. Can't update RM Delegation Token.); return; } try { updateRMDelegationTokenAndSequenceNumberInternal(rmDTIdentifier, renewDate, latestSequenceNumber); } catch (Exception e) { notifyStoreOperationFailed(e); } } {code} {code} @Override protected void updateStoredToken(RMDelegationTokenIdentifier id, long renewDate) { try { LOG.info(updating RMDelegation token with sequence number: + id.getSequenceNumber()); rmContext.getStateStore().updateRMDelegationTokenAndSequenceNumber(id, renewDate, id.getSequenceNumber()); } catch (Exception e) { LOG.error(Error in updating persisted RMDelegationToken with sequence number: + id.getSequenceNumber()); ExitUtil.terminate(1, e); } } {code} According to code above, even when renewing a DT, the last sequence number is updated in the store, which is wrong. For example, we have the following sequence: 1. Get DT 1 (seq = 1) 2. Get DT 2( seq = 2) 3. Renew DT 1 (seq = 1) 4. Restart RM The stored and then recovered last sequence number is 1. It makes the next created DT after RM restarting will conflict with DT 2 on sequence num. Second, the aforementioned bug doesn't happen actually, because the recovered last sequence num has been overwritten at by the correctly one. {code} public void recover(RMState rmState) throws Exception { LOG.info(recovering RMDelegationTokenSecretManager.); // recover RMDTMasterKeys for (DelegationKey dtKey : rmState.getRMDTSecretManagerState() .getMasterKeyState()) { addKey(dtKey); } // recover RMDelegationTokens MapRMDelegationTokenIdentifier, Long rmDelegationTokens = rmState.getRMDTSecretManagerState().getTokenState(); this.delegationTokenSequenceNumber = rmState.getRMDTSecretManagerState().getDTSequenceNumber(); for (Map.EntryRMDelegationTokenIdentifier, Long entry : rmDelegationTokens .entrySet()) { addPersistedDelegationToken(entry.getKey(), entry.getValue()); } } {code} The code above recovers delegationTokenSequenceNumber by reading the last sequence number in the store. It could be wrong. Fortunately, delegationTokenSequenceNumber updates it to the right number. {code} if (identifier.getSequenceNumber() getDelegationTokenSeqNum()) { setDelegationTokenSeqNum(identifier.getSequenceNumber()); } {code} All the stored identifiers will be gone through, and delegationTokenSequenceNumber will be set to the largest sequence number among these identifiers. Therefore, new DT will be assigned a sequence number which is always larger than that of all the recovered DT. To sum up, two negatives make a positive, but it's good to fix the issue. Please let me know if I've missed something here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245035#comment-14245035 ] Zhijie Shen commented on YARN-2837: --- bq. we need to keep track of the latest sequenceNumber so that it can be recovered. For the reason of YARN-2958, I don't think it's proper to following the way of RMStateStore. Please let me know if I missed something. Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2654) Revisit all shared cache config parameters to ensure quality names
[ https://issues.apache.org/jira/browse/YARN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo reassigned YARN-2654: -- Assignee: Chris Trezzo Revisit all shared cache config parameters to ensure quality names -- Key: YARN-2654 URL: https://issues.apache.org/jira/browse/YARN-2654 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Blocker Revisit all the shared cache config parameters in YarnConfiguration and yarn-default.xml to ensure quality names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2960) Add documentation for the YARN shared cache
Chris Trezzo created YARN-2960: -- Summary: Add documentation for the YARN shared cache Key: YARN-2960 URL: https://issues.apache.org/jira/browse/YARN-2960 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Add documentation around the architecture, api's and administration of the YARN shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[ https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated YARN-2950: -- Attachment: YARN-2950-2.patch Thanks Dustin! Looks good to me. I've gone ahead and made a small change to keep the line lengths less than 80 characters as per formatting requirements. Committing in a bit. Change message to mandate, not suggest JS requirement on UI --- Key: YARN-2950 URL: https://issues.apache.org/jira/browse/YARN-2950 Project: Hadoop YARN Issue Type: Improvement Components: webapp Affects Versions: 2.5.0 Reporter: Harsh J Assignee: Dustin Cote Priority: Minor Labels: newbie Attachments: YARN-2950-1.patch, YARN-2950-2.patch Most of YARN's UIs do not work with JavaScript disabled on the browser, cause they appear to send back data as JS arrays instead of within the actual HTML content. The JQueryUI prints only a mild warning about this suggesting that {{This page works best with javascript enabled.}}, when in fact it ought to be {{This page will not function without javascript enabled. Please enable javascript on your browser.}} or something as such (more direct). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2958) RMStateStore seems to unnecessarily and wronly store sequence number separately
[ https://issues.apache.org/jira/browse/YARN-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-2958: -- Assignee: Varun Saxena RMStateStore seems to unnecessarily and wronly store sequence number separately --- Key: YARN-2958 URL: https://issues.apache.org/jira/browse/YARN-2958 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Zhijie Shen Assignee: Varun Saxena It seems that RMStateStore updates last sequence number when storing or updating each individual DT, to recover the latest sequence number when RM restarting. First, the current logic seems to be problematic: {code} public synchronized void updateRMDelegationTokenAndSequenceNumber( RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate, int latestSequenceNumber) { if(isFencedState()) { LOG.info(State store is in Fenced state. Can't update RM Delegation Token.); return; } try { updateRMDelegationTokenAndSequenceNumberInternal(rmDTIdentifier, renewDate, latestSequenceNumber); } catch (Exception e) { notifyStoreOperationFailed(e); } } {code} {code} @Override protected void updateStoredToken(RMDelegationTokenIdentifier id, long renewDate) { try { LOG.info(updating RMDelegation token with sequence number: + id.getSequenceNumber()); rmContext.getStateStore().updateRMDelegationTokenAndSequenceNumber(id, renewDate, id.getSequenceNumber()); } catch (Exception e) { LOG.error(Error in updating persisted RMDelegationToken with sequence number: + id.getSequenceNumber()); ExitUtil.terminate(1, e); } } {code} According to code above, even when renewing a DT, the last sequence number is updated in the store, which is wrong. For example, we have the following sequence: 1. Get DT 1 (seq = 1) 2. Get DT 2( seq = 2) 3. Renew DT 1 (seq = 1) 4. Restart RM The stored and then recovered last sequence number is 1. It makes the next created DT after RM restarting will conflict with DT 2 on sequence num. Second, the aforementioned bug doesn't happen actually, because the recovered last sequence num has been overwritten at by the correctly one. {code} public void recover(RMState rmState) throws Exception { LOG.info(recovering RMDelegationTokenSecretManager.); // recover RMDTMasterKeys for (DelegationKey dtKey : rmState.getRMDTSecretManagerState() .getMasterKeyState()) { addKey(dtKey); } // recover RMDelegationTokens MapRMDelegationTokenIdentifier, Long rmDelegationTokens = rmState.getRMDTSecretManagerState().getTokenState(); this.delegationTokenSequenceNumber = rmState.getRMDTSecretManagerState().getDTSequenceNumber(); for (Map.EntryRMDelegationTokenIdentifier, Long entry : rmDelegationTokens .entrySet()) { addPersistedDelegationToken(entry.getKey(), entry.getValue()); } } {code} The code above recovers delegationTokenSequenceNumber by reading the last sequence number in the store. It could be wrong. Fortunately, delegationTokenSequenceNumber updates it to the right number. {code} if (identifier.getSequenceNumber() getDelegationTokenSeqNum()) { setDelegationTokenSeqNum(identifier.getSequenceNumber()); } {code} All the stored identifiers will be gone through, and delegationTokenSequenceNumber will be set to the largest sequence number among these identifiers. Therefore, new DT will be assigned a sequence number which is always larger than that of all the recovered DT. To sum up, two negatives make a positive, but it's good to fix the issue. Please let me know if I've missed something here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[ https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245101#comment-14245101 ] Hudson commented on YARN-2950: -- FAILURE: Integrated in Hadoop-trunk-Commit #6712 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6712/]) YARN-2950. Change message to mandate, not suggest JS requirement on UI. Contributed by Dustin Cote. (harsh: rev 0e37bbc8e3f8e96acd96522face2f4bb01584cb4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java * hadoop-yarn-project/CHANGES.txt Change message to mandate, not suggest JS requirement on UI --- Key: YARN-2950 URL: https://issues.apache.org/jira/browse/YARN-2950 Project: Hadoop YARN Issue Type: Improvement Components: webapp Affects Versions: 2.5.0 Reporter: Harsh J Assignee: Dustin Cote Priority: Minor Labels: newbie Fix For: 2.7.0 Attachments: YARN-2950-1.patch, YARN-2950-2.patch Most of YARN's UIs do not work with JavaScript disabled on the browser, cause they appear to send back data as JS arrays instead of within the actual HTML content. The JQueryUI prints only a mild warning about this suggesting that {{This page works best with javascript enabled.}}, when in fact it ought to be {{This page will not function without javascript enabled. Please enable javascript on your browser.}} or something as such (more direct). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245160#comment-14245160 ] Varun Saxena commented on YARN-2937: [~zjshen], bq. What's the problem around Random? There is a problem with Maths#abs. {{Maths.abs(Long.MIN_VALUE)==Long.MIN_VALUE}} i.e. Maths.abs doesnt work with lowest value of long. But Random may return that value. So changed Random to RandomUtils to give only positive values. Maths.abs can probably be removed in this case. Random has {{getInt}} method which can return positive values but no corresponding method for nextLong bq. Can we use IOUtils? Yes, will use it. bq. Is this removed because bufReader will consequently close fileReader too? No. This has been removed because instance of class {{FileReader}} has been removed and replaced with {{FileInputStream}} + {{InputStreamReader}}. This is because FileReader doesnt allow us to set Charset encoding. And yes, BufferedReader will close the underlying FileReader and FileReader will close the underlying stream. bq. Again, would you mind explaining a bit about what have been excluded in findbugs-exclude.xml? The exclusions are corresponding to *RV_RETURN_VALUE_IGNORED_BAD_PRACTICE*. Findbugs was complaining about return value of {{ThreadPoolExecutor#submit}} being ignored. This method returns a {{FutureT}} object. Generally {{Future#get}} is called to check the return value of executor. But this method is blocking and waits for thread execution to complete. If I call this, it would break the current behavior. Tried it as well and some test cases failed. Another alternative is to call {{Future#isDone}} but this most likely will always return false if you check it immediately. So I see no point in checking return value of call to submit. Hence, added it in exclusions. Findbugs was raising issue for below 2 lines. {code:title=ContainersLauncher.java:118|borderStyle=solid} containerLauncher.submit(launch); {code} {code:title=SharedCacheUploadService.java:118|borderStyle=solid} uploaderPool.submit(uploader); {code} Fix new findbugs warnings in hadoop-yarn-nodemanager Key: YARN-2937 URL: https://issues.apache.org/jira/browse/YARN-2937 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: HADOOP-11373.patch, YARN-2937.001.patch, YARN-2937.002.patch, YARN-2937.003.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245202#comment-14245202 ] Sunil G commented on YARN-2356: --- Hi Devaraj These findbugs problems are not induced by current JIRA. There were some rule changes done for java 7. I feel that caused the same. I cud open a new JIRA to fix this warning of PrintStream writer in all CLI classes. Kindly advice. Also test case failure is not related. yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at $Proxy12.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:428) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:76) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException): Application with id 'application_1402668848165_0015' doesn't exist in RM. -- This message was
[jira] [Commented] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245221#comment-14245221 ] Varun Saxena commented on YARN-2356: Yes [~sunilg] is correct. These findbugs will be handled by YARN-2940 yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at $Proxy12.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:428) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:76) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException): Application with id 'application_1402668848165_0015' doesn't exist in RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)