[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently
[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802783#comment-13802783 ] Hudson commented on YARN-1183: -- FAILURE: Integrated in Hadoop-Yarn-trunk #371 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/371/]) YARN-1183. MiniYARNCluster shutdown takes several minutes intermittently (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534800) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > MiniYARNCluster shutdown takes several minutes intermittently > - > > Key: YARN-1183 > URL: https://issues.apache.org/jira/browse/YARN-1183 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Andrey Klochkov >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.3.0 > > Attachments: YARN-1183--n2.patch, YARN-1183--n3.patch, > YARN-1183--n4.patch, YARN-1183--n5.patch, YARN-1183.patch > > > As described in MAPREDUCE-5501 sometimes M/R tests leave MRAppMaster java > processes living for several minutes after successful completion of the > corresponding test. There is a concurrency issue in MiniYARNCluster shutdown > logic which leads to this. Sometimes RM stops before an app master sends it's > last report, and then the app master keeps retrying for >6 minutes. In some > cases it leads to failures in subsequent tests, and it affects performance of > tests as app masters eat resources. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1305) RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException
[
https://issues.apache.org/jira/browse/YARN-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802780#comment-13802780
]
Hudson commented on YARN-1305:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #371 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/371/])
YARN-1305. RMHAProtocolService#serviceInit should handle HAUtil's
IllegalArgumentException (Tsuyoshi Ozawa via bikas) (bikas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534884)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestHAUtil.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMHAProtocolService.java
> RMHAProtocolService#serviceInit should handle HAUtil's
> IllegalArgumentException
> ---
>
> Key: YARN-1305
> URL: https://issues.apache.org/jira/browse/YARN-1305
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.2.1
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Labels: ha
> Fix For: 2.3.0
>
> Attachments: YARN-1305.1.patch, YARN-1305.2.patch, YARN-1305.3.patch,
> YARN-1305.4.patch, YARN-1305.5.patch, YARN-1305.6.patch, YARN-1305.7.patch,
> YARN-1305.8-2.patch, YARN-1305.8.patch, YARN-1305.9.patch
>
>
> When yarn.resourcemanager.ha.enabled is true, RMHAProtocolService#serviceInit
> calls HAUtil.setAllRpcAddresses. If the configuration values are null, it
> just throws IllegalArgumentException.
> It's messy to analyse which keys are null, so we should handle it and log the
> name of keys which are null.
> A current log dump is as follows:
> {code}
> 2013-10-15 06:24:53,431 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered
> UNIX signal handlers for [TERM, HUP, INT]
> 2013-10-15 06:24:54,203 INFO org.apache.hadoop.service.AbstractService:
> Service RMHAProtocolService failed in state INITED; cause:
> java.lang.IllegalArgumentException: Property value must not be null
> java.lang.IllegalArgumentException: Property value must not be null
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:816)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:798)
> at org.apache.hadoop.yarn.conf.HAUtil.setConfValue(HAUtil.java:100)
> at
> org.apache.hadoop.yarn.conf.HAUtil.setAllRpcAddresses(HAUtil.java:105)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceInit(RMHAProtocolService.java:60)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:187)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:940)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1330) Fair Scheduler: defaultQueueSchedulingPolicy does not take effect
[ https://issues.apache.org/jira/browse/YARN-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802785#comment-13802785 ] Hudson commented on YARN-1330: -- FAILURE: Integrated in Hadoop-Yarn-trunk #371 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/371/]) YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534861) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Fair Scheduler: defaultQueueSchedulingPolicy does not take effect > - > > Key: YARN-1330 > URL: https://issues.apache.org/jira/browse/YARN-1330 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1330-1.patch, YARN-1330-1.patch, YARN-1330.patch > > > The defaultQueueSchedulingPolicy property for the Fair Scheduler allocations > file doesn't take effect. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1305) RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException
[
https://issues.apache.org/jira/browse/YARN-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802814#comment-13802814
]
Hudson commented on YARN-1305:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1561 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1561/])
YARN-1305. RMHAProtocolService#serviceInit should handle HAUtil's
IllegalArgumentException (Tsuyoshi Ozawa via bikas) (bikas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534884)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestHAUtil.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMHAProtocolService.java
> RMHAProtocolService#serviceInit should handle HAUtil's
> IllegalArgumentException
> ---
>
> Key: YARN-1305
> URL: https://issues.apache.org/jira/browse/YARN-1305
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.2.1
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Labels: ha
> Fix For: 2.3.0
>
> Attachments: YARN-1305.1.patch, YARN-1305.2.patch, YARN-1305.3.patch,
> YARN-1305.4.patch, YARN-1305.5.patch, YARN-1305.6.patch, YARN-1305.7.patch,
> YARN-1305.8-2.patch, YARN-1305.8.patch, YARN-1305.9.patch
>
>
> When yarn.resourcemanager.ha.enabled is true, RMHAProtocolService#serviceInit
> calls HAUtil.setAllRpcAddresses. If the configuration values are null, it
> just throws IllegalArgumentException.
> It's messy to analyse which keys are null, so we should handle it and log the
> name of keys which are null.
> A current log dump is as follows:
> {code}
> 2013-10-15 06:24:53,431 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered
> UNIX signal handlers for [TERM, HUP, INT]
> 2013-10-15 06:24:54,203 INFO org.apache.hadoop.service.AbstractService:
> Service RMHAProtocolService failed in state INITED; cause:
> java.lang.IllegalArgumentException: Property value must not be null
> java.lang.IllegalArgumentException: Property value must not be null
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:816)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:798)
> at org.apache.hadoop.yarn.conf.HAUtil.setConfValue(HAUtil.java:100)
> at
> org.apache.hadoop.yarn.conf.HAUtil.setAllRpcAddresses(HAUtil.java:105)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceInit(RMHAProtocolService.java:60)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:187)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:940)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1330) Fair Scheduler: defaultQueueSchedulingPolicy does not take effect
[ https://issues.apache.org/jira/browse/YARN-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802819#comment-13802819 ] Hudson commented on YARN-1330: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1561 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1561/]) YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534861) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Fair Scheduler: defaultQueueSchedulingPolicy does not take effect > - > > Key: YARN-1330 > URL: https://issues.apache.org/jira/browse/YARN-1330 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1330-1.patch, YARN-1330-1.patch, YARN-1330.patch > > > The defaultQueueSchedulingPolicy property for the Fair Scheduler allocations > file doesn't take effect. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently
[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802817#comment-13802817 ] Hudson commented on YARN-1183: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1561 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1561/]) YARN-1183. MiniYARNCluster shutdown takes several minutes intermittently (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534800) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > MiniYARNCluster shutdown takes several minutes intermittently > - > > Key: YARN-1183 > URL: https://issues.apache.org/jira/browse/YARN-1183 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Andrey Klochkov >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.3.0 > > Attachments: YARN-1183--n2.patch, YARN-1183--n3.patch, > YARN-1183--n4.patch, YARN-1183--n5.patch, YARN-1183.patch > > > As described in MAPREDUCE-5501 sometimes M/R tests leave MRAppMaster java > processes living for several minutes after successful completion of the > corresponding test. There is a concurrency issue in MiniYARNCluster shutdown > logic which leads to this. Sometimes RM stops before an app master sends it's > last report, and then the app master keeps retrying for >6 minutes. In some > cases it leads to failures in subsequent tests, and it affects performance of > tests as app masters eat resources. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-671) Add an interface on the RM to move NMs into a maintenance state
[ https://issues.apache.org/jira/browse/YARN-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802878#comment-13802878 ] Jason Lowe commented on YARN-671: - Note that simply waiting for containers to complete doesn't address auxiliary services like MapReduce's ShuffleHandler that want to serve up data after containers have completed. If we simply wait until containers have completed and stop the NM immediately afterwards then we can cause fetch failures leading to a relaunch of map tasks that ran on that node. In that case we would have been better off killing the map task immediately, as the application would have recovered faster. Therefore without more intimate knowledge of what an auxiliary service is really doing with respect to the containers that ran on a node, we may have to generalize the timeout to all applications have completed that have run at least one container on the node. > Add an interface on the RM to move NMs into a maintenance state > --- > > Key: YARN-671 > URL: https://issues.apache.org/jira/browse/YARN-671 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1305) RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException
[
https://issues.apache.org/jira/browse/YARN-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802880#comment-13802880
]
Hudson commented on YARN-1305:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1587 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1587/])
YARN-1305. RMHAProtocolService#serviceInit should handle HAUtil's
IllegalArgumentException (Tsuyoshi Ozawa via bikas) (bikas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534884)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestHAUtil.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMHAProtocolService.java
> RMHAProtocolService#serviceInit should handle HAUtil's
> IllegalArgumentException
> ---
>
> Key: YARN-1305
> URL: https://issues.apache.org/jira/browse/YARN-1305
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.2.1
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Labels: ha
> Fix For: 2.3.0
>
> Attachments: YARN-1305.1.patch, YARN-1305.2.patch, YARN-1305.3.patch,
> YARN-1305.4.patch, YARN-1305.5.patch, YARN-1305.6.patch, YARN-1305.7.patch,
> YARN-1305.8-2.patch, YARN-1305.8.patch, YARN-1305.9.patch
>
>
> When yarn.resourcemanager.ha.enabled is true, RMHAProtocolService#serviceInit
> calls HAUtil.setAllRpcAddresses. If the configuration values are null, it
> just throws IllegalArgumentException.
> It's messy to analyse which keys are null, so we should handle it and log the
> name of keys which are null.
> A current log dump is as follows:
> {code}
> 2013-10-15 06:24:53,431 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered
> UNIX signal handlers for [TERM, HUP, INT]
> 2013-10-15 06:24:54,203 INFO org.apache.hadoop.service.AbstractService:
> Service RMHAProtocolService failed in state INITED; cause:
> java.lang.IllegalArgumentException: Property value must not be null
> java.lang.IllegalArgumentException: Property value must not be null
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:816)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:798)
> at org.apache.hadoop.yarn.conf.HAUtil.setConfValue(HAUtil.java:100)
> at
> org.apache.hadoop.yarn.conf.HAUtil.setAllRpcAddresses(HAUtil.java:105)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceInit(RMHAProtocolService.java:60)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:187)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:940)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1183) MiniYARNCluster shutdown takes several minutes intermittently
[ https://issues.apache.org/jira/browse/YARN-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802883#comment-13802883 ] Hudson commented on YARN-1183: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1587 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1587/]) YARN-1183. MiniYARNCluster shutdown takes several minutes intermittently (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534800) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > MiniYARNCluster shutdown takes several minutes intermittently > - > > Key: YARN-1183 > URL: https://issues.apache.org/jira/browse/YARN-1183 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Andrey Klochkov >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.3.0 > > Attachments: YARN-1183--n2.patch, YARN-1183--n3.patch, > YARN-1183--n4.patch, YARN-1183--n5.patch, YARN-1183.patch > > > As described in MAPREDUCE-5501 sometimes M/R tests leave MRAppMaster java > processes living for several minutes after successful completion of the > corresponding test. There is a concurrency issue in MiniYARNCluster shutdown > logic which leads to this. Sometimes RM stops before an app master sends it's > last report, and then the app master keeps retrying for >6 minutes. In some > cases it leads to failures in subsequent tests, and it affects performance of > tests as app masters eat resources. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1330) Fair Scheduler: defaultQueueSchedulingPolicy does not take effect
[ https://issues.apache.org/jira/browse/YARN-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802885#comment-13802885 ] Hudson commented on YARN-1330: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1587 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1587/]) YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1534861) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Fair Scheduler: defaultQueueSchedulingPolicy does not take effect > - > > Key: YARN-1330 > URL: https://issues.apache.org/jira/browse/YARN-1330 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.2.1 > > Attachments: YARN-1330-1.patch, YARN-1330-1.patch, YARN-1330.patch > > > The defaultQueueSchedulingPolicy property for the Fair Scheduler allocations > file doesn't take effect. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-671) Add an interface on the RM to move NMs into a maintenance state
[ https://issues.apache.org/jira/browse/YARN-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802888#comment-13802888 ] Jason Lowe commented on YARN-671: - Another concern is handling of the available resources for the scheduler and reported headroom to AMs. Are the total resources of the cluster adjusted downwards as containers on the node complete, or does the cluster run "above capacity" while we're waiting for the remaining containers? In that sense this seems closely related to YARN-914. > Add an interface on the RM to move NMs into a maintenance state > --- > > Key: YARN-671 > URL: https://issues.apache.org/jira/browse/YARN-671 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1336) Work-preserving nodemanager restart
[ https://issues.apache.org/jira/browse/YARN-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803034#comment-13803034 ] Cindy Li commented on YARN-1336: I'm interested in working on this. I'm working on another related Jira: Yarn-671 https://issues.apache.org/jira/browse/YARN-671 > Work-preserving nodemanager restart > --- > > Key: YARN-1336 > URL: https://issues.apache.org/jira/browse/YARN-1336 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Jason Lowe > > This serves as an umbrella ticket for tasks related to work-preserving > nodemanager restart. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-451) Add more metrics to RM page
[
https://issues.apache.org/jira/browse/YARN-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803113#comment-13803113
]
Sangjin Lee commented on YARN-451:
--
Forgot to answer this:
{quote}
I took a look at the scheduler page, and all I see for current allocation is
per-user-per-queue and not per app. Where are you seeing the current assignment
for each app on the scheduler page?
{quote}
I think what Joep is referring to is the fair share numbers in case of the fair
scheduler, and whatever corresponds to those numbers in the capacity scheduler
page.
> Add more metrics to RM page
> ---
>
> Key: YARN-451
> URL: https://issues.apache.org/jira/browse/YARN-451
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.0.3-alpha
>Reporter: Lohit Vijayarenu
>Assignee: Sangjin Lee
>Priority: Blocker
> Attachments: in_progress_2x.png, yarn-451-trunk-20130916.1.patch
>
>
> ResourceManager webUI shows list of RUNNING applications, but it does not
> tell which applications are requesting more resource compared to others. With
> cluster running hundreds of applications at once it would be useful to have
> some kind of metric to show high-resource usage applications vs low-resource
> usage ones. At the minimum showing number of containers is good option.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-975) Add a file-system implementation for history-storage
[
https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803124#comment-13803124
]
Mayank Bansal commented on YARN-975:
Thanks for [~zjshen] for the patch.
I still see this in the latest patch
catch (IOException e) {
+// Eat the exception not to disturb the getting the next
+// ApplicationHistoryData
+historyDataMap.put(appId, null);
+ }
> Add a file-system implementation for history-storage
>
>
> Key: YARN-975
> URL: https://issues.apache.org/jira/browse/YARN-975
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-975.1.patch, YARN-975.2.patch, YARN-975.3.patch,
> YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch, YARN-975.7.patch,
> YARN-975.8.patch
>
>
> HDFS implementation should be a standard persistence strategy of history
> storage
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1307) Rethink znode structure for RM HA
[
https://issues.apache.org/jira/browse/YARN-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803148#comment-13803148
]
Bikas Saha commented on YARN-1307:
--
It is fine to store a batch of delegation token in the same znode. In fact that
may be better than creating a huge number of znode. A setData operation is easy
to do on a znode. So the stateStore can keep the current (upto 500kb) batch in
memory. When a new delegation token arrives then the store can add it to the
batch and overwrite the znode data with the new batch. Keep doing it till the
batch size is ~500kb. When the data size exceeds the size of a batch then a new
znode is created.
There is no need to encode the sequence number in the name. It can be made part
of the znode data. The sequence number znode stores the sequence number of the
tokens. We encoded the sequence number in the name for HDFS so that we dont
have to go to the data nodes to write the data. this is not necessary in ZK
since it supports a setData operation that overwrites the previous data. We can
simply name the znode something like RMDelegationTokenSequenceNumber.
> Rethink znode structure for RM HA
> -
>
> Key: YARN-1307
> URL: https://issues.apache.org/jira/browse/YARN-1307
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-1307.1.patch, YARN-1307.2.patch
>
>
> Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659,
> YARN-1222). The motivation of this JIRA is quoted from Bikas' comment in
> YARN-1222:
> {quote}
> We should move to creating a node hierarchy for apps such that all znodes for
> an app are stored under an app znode instead of the app root znode. This will
> help in removeApplication and also in scaling better on ZK. The earlier code
> was written this way to ensure create/delete happens under a root znode for
> fencing. But given that we have moved to multi-operations globally, this isnt
> required anymore.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-975) Add a file-system implementation for history-storage
[ https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-975: - Attachment: YARN-975.9.patch Addressed Mayank's comment. In addition, changed the permission value from decimal to octal to be more readable. > Add a file-system implementation for history-storage > > > Key: YARN-975 > URL: https://issues.apache.org/jira/browse/YARN-975 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-975.1.patch, YARN-975.2.patch, YARN-975.3.patch, > YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch, YARN-975.7.patch, > YARN-975.8.patch, YARN-975.9.patch > > > HDFS implementation should be a standard persistence strategy of history > storage -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-975) Add a file-system implementation for history-storage
[
https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803192#comment-13803192
]
Hadoop QA commented on YARN-975:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609901/YARN-975.9.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2264//console
This message is automatically generated.
> Add a file-system implementation for history-storage
>
>
> Key: YARN-975
> URL: https://issues.apache.org/jira/browse/YARN-975
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-975.1.patch, YARN-975.2.patch, YARN-975.3.patch,
> YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch, YARN-975.7.patch,
> YARN-975.8.patch, YARN-975.9.patch
>
>
> HDFS implementation should be a standard persistence strategy of history
> storage
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-975) Add a file-system implementation for history-storage
[ https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803200#comment-13803200 ] Mayank Bansal commented on YARN-975: Looks good +1 Thanks, Mayank > Add a file-system implementation for history-storage > > > Key: YARN-975 > URL: https://issues.apache.org/jira/browse/YARN-975 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-975.1.patch, YARN-975.2.patch, YARN-975.3.patch, > YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch, YARN-975.7.patch, > YARN-975.8.patch, YARN-975.9.patch > > > HDFS implementation should be a standard persistence strategy of history > storage -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-956) [YARN-321] Add a testable in-memory HistoryStorage
[ https://issues.apache.org/jira/browse/YARN-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803245#comment-13803245 ] Vinod Kumar Vavilapalli commented on YARN-956: -- Quick comments on the patch: - Don't like the statics. We are eventually going to run multiple tests in parallel and statics in InMemoryHistoryStorage will bite us. - If we get rid of the static data, testWriteApplicationHistoryAgain(), classTearDown(), "+ //The order of the test cases matters" etc are useless? - "is already presented" -> "is already stored" everywhere. - TestApplicationHistoryStore: Rename it to ApplicationHistoryStoreTestUtils? > [YARN-321] Add a testable in-memory HistoryStorage > --- > > Key: YARN-956 > URL: https://issues.apache.org/jira/browse/YARN-956 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-956-1.patch, YARN-956-2.patch, YARN-956-3.patch, > YARN-956.4.patch, YARN-956.5.patch, YARN-956.6.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803262#comment-13803262 ] Vinod Kumar Vavilapalli commented on YARN-947: -- Quick comments on the patch: - We don't need a new YarnContainerState, already have ContainerState. It can be argued that ContainerState should have more information. In that case, we should directly change ContainerState itself - but have to think about compatibility. Do we ever expose ContainerState in any existing APIs in 2.2.0? In any case I'd like to postpone that discussion into a separate JIRA and use ContainerState itself for now. - We in future may need a separate ApplicationAttemptStartDataProto and ApplicationAttemptRegisteredDataProto. For now, we can just store both of them after AM registration, but that increases the possibility of loss of information on RM restart. May be another JIRA. - We will need a APP_ATTEMPT_FINAL_SAVING state as being discussed on YARN-891. Please add it. - I think we should separate client facing records from the event records. For e.g. ContainerHistoryData.java shouldn't be in server package. Again another JIRA, but this one's more important. - SchedulerUtils.java changes are unnecessary. > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (YARN-674) Slow or failing DelegationToken renewals on submission itself make RM unavailable
[ https://issues.apache.org/jira/browse/YARN-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-674: -- Assignee: Omkar Vinit Joshi (was: Vinod Kumar Vavilapalli) > Slow or failing DelegationToken renewals on submission itself make RM > unavailable > - > > Key: YARN-674 > URL: https://issues.apache.org/jira/browse/YARN-674 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Omkar Vinit Joshi > > This was caused by YARN-280. A slow or a down NameNode for will make it look > like RM is unavailable as it may run out of RPC handlers due to blocked > client submissions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-674) Slow or failing DelegationToken renewals on submission itself make RM unavailable
[ https://issues.apache.org/jira/browse/YARN-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803278#comment-13803278 ] Omkar Vinit Joshi commented on YARN-674: Taking it over.. > Slow or failing DelegationToken renewals on submission itself make RM > unavailable > - > > Key: YARN-674 > URL: https://issues.apache.org/jira/browse/YARN-674 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > This was caused by YARN-280. A slow or a down NameNode for will make it look > like RM is unavailable as it may run out of RPC handlers due to blocked > client submissions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803289#comment-13803289 ] Hitesh Shah commented on YARN-1303: --- [~xgong] Have you looked at fixing the underlying issue of passing the args correctly from the client to the AM? At the moment, they are sent via the AM's command line which can create problems. To pass to the AM via command-line requires correct escaping based on type of OS. Or, they could be passed in through a file which would hide away such aspects. > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.2.1 > > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803330#comment-13803330 ] Xuan Gong commented on YARN-1303: - Yes, not escaping and quoting the command line args are the issue. To fix it, either we can expect that the client will give the correct command with the correct escaping and quoting (That will not alway be happened.) or we can add double quote or single quote with the correct escape character (such as \, etc) to modify the command in Client and AppMaster code. But this will need the codes know how and when to add escaping and quoting based on the command. It is really difficult to cover all the cases. So I think that passed in through a file might be the simplest way to solve it. Tell the client: please create your own shell script if you want to run some complex commands. [~vinodkv] Could you give me further comments ? > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.2.1 > > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
Alejandro Abdelnur created YARN-1343: Summary: NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs Key: YARN-1343 URL: https://issues.apache.org/jira/browse/YARN-1343 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.2.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Critical Fix For: 2.2.1 If a NodeManager joins the cluster or gets restarted, running AMs never receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-956) [YARN-321] Add a testable in-memory HistoryStorage
[ https://issues.apache.org/jira/browse/YARN-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803340#comment-13803340 ] Mayank Bansal commented on YARN-956: Thanks [~vinodkv] for the review. Earlier I had the same discussion with [~zjshen] offline and I pointed the same point for static variables. Zhijie convinced me for static saying that we should only have one store per system. I think there is a point to that because if somebody just wants to use the memory store for some reason for AHS then it would be very hard for us to support. Else we have to tell them memory store is just for testing and if they want to use something like that then they have to write their own store. Thanks, Mayank > [YARN-321] Add a testable in-memory HistoryStorage > --- > > Key: YARN-956 > URL: https://issues.apache.org/jira/browse/YARN-956 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-956-1.patch, YARN-956-2.patch, YARN-956-3.patch, > YARN-956.4.patch, YARN-956.5.patch, YARN-956.6.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803337#comment-13803337 ] Alejandro Abdelnur commented on YARN-1343: -- The problem is that the RMNodeImpl.AddNodeTransition() is not dispatching a NodesListManagerEventType.NODE_USABLE event to the NodeListManager. Plus, the NodeListManager handle() for NODE_UNUSABLE should dispatch RMAppNodeUpdateType.NODE_USABLE events to the apps even if the node was not in the unusable list. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-891) Store completed application information in RM state store
[ https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-891: - Attachment: YARN-891.3.patch > Store completed application information in RM state store > - > > Key: YARN-891 > URL: https://issues.apache.org/jira/browse/YARN-891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch, > YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch, > YARN-891.patch, YARN-891.patch > > > Store completed application/attempt info in RMStateStore when > application/attempt completes. This solves some problems like finished > application get lost after RM restart and some other races like YARN-1195 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803391#comment-13803391 ] Vinod Kumar Vavilapalli commented on YARN-947: -- Turns out ContainerHistoryData is a server side record that is used only by the reader. In that case, we can keep it in the server package but I don't think it should be a pb implementation. Thinking beyond, may be we should ditch AppilcationHistoryData etc completely and simply use ApplicationReport, ApplicationAttemptReport etc in the Reader. Thoughts? > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated YARN-1343: - Attachment: YARN-1343.patch I've also verified in a deployment the AM does received the node update. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-891) Store completed application information in RM state store
[ https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803437#comment-13803437 ] Jian He commented on YARN-891: -- Had an offline discussion with Vinod and made a bunch of changes, mainly: - create new updateApplicationStateInternal API of RMStateStore for FS/ZK state to override for updating application state and correspondingly the update events. - refactor and rename some newly added methods/transitions inside RMAppImpl and RMAppAttemptImpl - RMAppManager.recover() is changed to always recover applications, let RMAppImpl transition internally decide whether to launch the application or not. - Add more unit tests in TestRMRestart for getting applications report / list after RM restarts. - Add test for FS/ZK state store to verify newly added fields are persisted well. To do: - We should move the newInstance methods from both the data PM impls to the data objects themselves. - Single node test with ZK store. > Store completed application information in RM state store > - > > Key: YARN-891 > URL: https://issues.apache.org/jira/browse/YARN-891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch, > YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch, > YARN-891.patch, YARN-891.patch > > > Store completed application/attempt info in RMStateStore when > application/attempt completes. This solves some problems like finished > application get lost after RM restart and some other races like YARN-1195 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803442#comment-13803442 ] Sandy Ryza commented on YARN-1343: -- +1 > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803446#comment-13803446 ] Sandy Ryza commented on YARN-1343: -- pending jenkins > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803458#comment-13803458 ] Mayank Bansal commented on YARN-947: [~zjshen] I think we should have seprate store implemetation to clients facing that way we can change whenever we need to at store level. > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1303) Allow multiple commands separating with ";" in distributed-shell
[ https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803463#comment-13803463 ] Hitesh Shah commented on YARN-1303: --- [~xgong] How about ensuring that the data passed into the client by the user is correctly passed to the AM and to the launch script in the task. As for escaping, we can rely on the user to correctly escape chars as needed. For example, if a user wants to run "echo $FOO" , the user should do "echo \$FOO" to ensure that the final command run has "echo $FOO" > Allow multiple commands separating with ";" in distributed-shell > > > Key: YARN-1303 > URL: https://issues.apache.org/jira/browse/YARN-1303 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Reporter: Tassapol Athiapinya >Assignee: Xuan Gong > Fix For: 2.2.1 > > Attachments: YARN-1303.1.patch, YARN-1303.2.patch, YARN-1303.3.patch, > YARN-1303.3.patch, YARN-1303.4.patch, YARN-1303.4.patch, YARN-1303.5.patch, > YARN-1303.6.patch, YARN-1303.7.patch > > > In shell, we can do "ls; ls" to run 2 commands at once. > In distributed shell, this is not working. We should improve to allow this to > occur. There are practical use cases that I know of to run multiple commands > or to set environment variables before a command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-891) Store completed application information in RM state store
[
https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803469#comment-13803469
]
Hadoop QA commented on YARN-891:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609935/YARN-891.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 2
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2265//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2265//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2265//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2265//console
This message is automatically generated.
> Store completed application information in RM state store
> -
>
> Key: YARN-891
> URL: https://issues.apache.org/jira/browse/YARN-891
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch,
> YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch,
> YARN-891.patch, YARN-891.patch
>
>
> Store completed application/attempt info in RMStateStore when
> application/attempt completes. This solves some problems like finished
> application get lost after RM restart and some other races like YARN-1195
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-311: Attachment: YARN-311-v9.patch Sync up with latest trunk in v9 patch. > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, > YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, > YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[
https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803478#comment-13803478
]
Hadoop QA commented on YARN-1343:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609939/YARN-1343.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2266//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2266//console
This message is automatically generated.
> NodeManagers additions/restarts are not reported as node updates in
> AllocateResponse responses to AMs
> -
>
> Key: YARN-1343
> URL: https://issues.apache.org/jira/browse/YARN-1343
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.2.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>Priority: Critical
> Fix For: 2.2.1
>
> Attachments: YARN-1343.patch
>
>
> If a NodeManager joins the cluster or gets restarted, running AMs never
> receive the node update indicating the Node is running.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1163) Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
[ https://issues.apache.org/jira/browse/YARN-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-1163: - Attachment: YARN-1163-v2.patch Sync up patch with latest trunk. > Cleanup code for AssignMapsWithLocality() in RMContainerAllocator > - > > Key: YARN-1163 > URL: https://issues.apache.org/jira/browse/YARN-1163 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: YARN-1163-v1.patch, YARN-1163-v2.patch > > > In RMContainerAllocator, AssignMapsWithLocality() is a very important method > to assign map tasks on allocated containers with conforming different level > of locality (dataLocal, rackLocal, etc.). However, this method messed with > different code logic to handle different type of locality but have lots of > similar behaviours. This is hard to maintain as well as do extension with > other locality type, so we need some more clear code here. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[
https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803489#comment-13803489
]
Bikas Saha commented on YARN-1343:
--
If the node was marked as unhealthy then it should have been in
unusableRMNodesConcurrentSet. So when it becomes healthy it must be found in
that list. Why are we moving the code outside the check?
{code}
if (unusableRMNodesConcurrentSet.contains(eventNode)) {
LOG.debug(eventNode + " reported usable");
unusableRMNodesConcurrentSet.remove(eventNode);
-for (RMApp app : rmContext.getRMApps().values()) {
- this.rmContext
- .getDispatcher()
- .getEventHandler()
- .handle(
- new RMAppNodeUpdateEvent(app.getApplicationId(), eventNode,
- RMAppNodeUpdateType.NODE_USABLE));
-}
{code}
> NodeManagers additions/restarts are not reported as node updates in
> AllocateResponse responses to AMs
> -
>
> Key: YARN-1343
> URL: https://issues.apache.org/jira/browse/YARN-1343
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.2.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>Priority: Critical
> Fix For: 2.2.1
>
> Attachments: YARN-1343.patch
>
>
> If a NodeManager joins the cluster or gets restarted, running AMs never
> receive the node update indicating the Node is running.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803493#comment-13803493 ] Alejandro Abdelnur commented on YARN-1343: -- because if not, added/restarted nodes never make it to AMs via node updates as they are not in the unusable list. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-891) Store completed application information in RM state store
[ https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-891: - Attachment: YARN-891.4.patch New patch fixed the warnings > Store completed application information in RM state store > - > > Key: YARN-891 > URL: https://issues.apache.org/jira/browse/YARN-891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch, > YARN-891.4.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch, > YARN-891.patch, YARN-891.patch, YARN-891.patch > > > Store completed application/attempt info in RMStateStore when > application/attempt completes. This solves some problems like finished > application get lost after RM restart and some other races like YARN-1195 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803514#comment-13803514 ] Bikas Saha commented on YARN-1343: -- Should we create a NODE_ADDED instead of overloading NODE_USABLE in case downstream consumers need to differentiate the cases. The update type enum (eg. NODE_USABLE) is passed in the event sent to the consumers (in this case RMApp). Reconnect of a NM after restart has 2 cases, it comes back with identical state/resources. Does the app need to know about this? It comes back with different state. Does the app need to know about this? If the answer is yes to both then do the 2 situations need to be differentiated? > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1163) Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803516#comment-13803516
]
Hadoop QA commented on YARN-1163:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609947/YARN-1163-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2267//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2267//console
This message is automatically generated.
> Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
> -
>
> Key: YARN-1163
> URL: https://issues.apache.org/jira/browse/YARN-1163
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Minor
> Attachments: YARN-1163-v1.patch, YARN-1163-v2.patch
>
>
> In RMContainerAllocator, AssignMapsWithLocality() is a very important method
> to assign map tasks on allocated containers with conforming different level
> of locality (dataLocal, rackLocal, etc.). However, this method messed with
> different code logic to handle different type of locality but have lots of
> similar behaviours. This is hard to maintain as well as do extension with
> other locality type, so we need some more clear code here.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[
https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803522#comment-13803522
]
Hadoop QA commented on YARN-311:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609943/YARN-311-v9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2268//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2268//console
This message is automatically generated.
> Dynamic node resource configuration: core scheduler changes
> ---
>
> Key: YARN-311
> URL: https://issues.apache.org/jira/browse/YARN-311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager, scheduler
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch,
> YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch,
> YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch,
> YARN-311-v9.patch
>
>
> As the first step, we go for resource change on RM side and expose admin APIs
> (admin protocol, CLI, REST and JMX API) later. In this jira, we will only
> contain changes in scheduler.
> The flow to update node's resource and awareness in resource scheduling is:
> 1. Resource update is through admin API to RM and take effect on RMNodeImpl.
> 2. When next NM heartbeat for updating status comes, the RMNode's resource
> change will be aware and the delta resource is added to schedulerNode's
> availableResource before actual scheduling happens.
> 3. Scheduler do resource allocation according to new availableResource in
> SchedulerNode.
> For more design details, please refer proposal and discussions in parent
> JIRA: YARN-291.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1163) Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
[ https://issues.apache.org/jira/browse/YARN-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-1163: - Issue Type: Improvement (was: Sub-task) Parent: (was: YARN-18) > Cleanup code for AssignMapsWithLocality() in RMContainerAllocator > - > > Key: YARN-1163 > URL: https://issues.apache.org/jira/browse/YARN-1163 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: YARN-1163-v1.patch, YARN-1163-v2.patch > > > In RMContainerAllocator, AssignMapsWithLocality() is a very important method > to assign map tasks on allocated containers with conforming different level > of locality (dataLocal, rackLocal, etc.). However, this method messed with > different code logic to handle different type of locality but have lots of > similar behaviours. This is hard to maintain as well as do extension with > other locality type, so we need some more clear code here. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1323) Set HTTPS webapp address along with other RPC addresses
[ https://issues.apache.org/jira/browse/YARN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1323: --- Attachment: yarn-1323-1.patch Trivial patch that adds handling webapp.https.address. > Set HTTPS webapp address along with other RPC addresses > --- > > Key: YARN-1323 > URL: https://issues.apache.org/jira/browse/YARN-1323 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1323-1.patch > > > YARN-1232 adds the ability to configure multiple RMs, but missed out the > https web app address. Need to add that in. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1323) Set HTTPS webapp address along with other RPC addresses
[ https://issues.apache.org/jira/browse/YARN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803531#comment-13803531 ] Karthik Kambatla commented on YARN-1323: TestHAUtil tests this as well. > Set HTTPS webapp address along with other RPC addresses > --- > > Key: YARN-1323 > URL: https://issues.apache.org/jira/browse/YARN-1323 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1323-1.patch > > > YARN-1232 adds the ability to configure multiple RMs, but missed out the > https web app address. Need to add that in. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-311) Dynamic node resource configuration: core scheduler changes
[ https://issues.apache.org/jira/browse/YARN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803534#comment-13803534 ] Junping Du commented on YARN-311: - Hi [~vinodkv] and [~vicaya], would you help to review it again? Thx! > Dynamic node resource configuration: core scheduler changes > --- > > Key: YARN-311 > URL: https://issues.apache.org/jira/browse/YARN-311 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-311-v1.patch, YARN-311-v2.patch, YARN-311-v3.patch, > YARN-311-v4.patch, YARN-311-v4.patch, YARN-311-v5.patch, YARN-311-v6.1.patch, > YARN-311-v6.2.patch, YARN-311-v6.patch, YARN-311-v7.patch, YARN-311-v8.patch, > YARN-311-v9.patch > > > As the first step, we go for resource change on RM side and expose admin APIs > (admin protocol, CLI, REST and JMX API) later. In this jira, we will only > contain changes in scheduler. > The flow to update node's resource and awareness in resource scheduling is: > 1. Resource update is through admin API to RM and take effect on RMNodeImpl. > 2. When next NM heartbeat for updating status comes, the RMNode's resource > change will be aware and the delta resource is added to schedulerNode's > availableResource before actual scheduling happens. > 3. Scheduler do resource allocation according to new availableResource in > SchedulerNode. > For more design details, please refer proposal and discussions in parent > JIRA: YARN-291. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803542#comment-13803542 ] Zhijie Shen commented on YARN-947: -- Thanks for the comments. Having discussion with [~vinodkv] and [~mayank_bansal]. Here's some update: 1. Remove PBImpl, and change HistoryData to a simple object that containing of the fields. Discuss the issue of changing readers to Report directly in a follow up ticket. 2. Now we can only modify the yarn_protos and YarnApplicationAttemptState to add the new state enum. Need to file a follow up ticket to update the conversion method in RMServerUtils whenever YARN-891 is done > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1323) Set HTTPS webapp address along with other RPC addresses
[
https://issues.apache.org/jira/browse/YARN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803548#comment-13803548
]
Hadoop QA commented on YARN-1323:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609956/yarn-1323-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2270//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2270//console
This message is automatically generated.
> Set HTTPS webapp address along with other RPC addresses
> ---
>
> Key: YARN-1323
> URL: https://issues.apache.org/jira/browse/YARN-1323
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.3.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: ha
> Attachments: yarn-1323-1.patch
>
>
> YARN-1232 adds the ability to configure multiple RMs, but missed out the
> https web app address. Need to add that in.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1323) Set HTTPS webapp address along with other RPC addresses
[ https://issues.apache.org/jira/browse/YARN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803556#comment-13803556 ] Karthik Kambatla commented on YARN-1323: TestHAUtil already tests this - hence, didn't add any new tests. > Set HTTPS webapp address along with other RPC addresses > --- > > Key: YARN-1323 > URL: https://issues.apache.org/jira/browse/YARN-1323 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1323-1.patch > > > YARN-1232 adds the ability to configure multiple RMs, but missed out the > https web app address. Need to add that in. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1222) Make improvements in ZKRMStateStore for fencing
[
https://issues.apache.org/jira/browse/YARN-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803559#comment-13803559
]
Tsuyoshi OZAWA commented on YARN-1222:
--
{code}
Based on reading this patch YARN-1307 neither blocks this nor is blocked by it.
We can continue on that in parallel.
{code}
OK. Thank you for coordinating, Bikas.
> Make improvements in ZKRMStateStore for fencing
> ---
>
> Key: YARN-1222
> URL: https://issues.apache.org/jira/browse/YARN-1222
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Karthik Kambatla
> Attachments: yarn-1222-1.patch, yarn-1222-2.patch
>
>
> Using multi-operations for every ZK interaction.
> In every operation, automatically creating/deleting a lock znode that is the
> child of the root znode. This is to achieve fencing by modifying the
> create/delete permissions on the root znode.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-891) Store completed application information in RM state store
[
https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803558#comment-13803558
]
Hadoop QA commented on YARN-891:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609954/YARN-891.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2269//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2269//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2269//console
This message is automatically generated.
> Store completed application information in RM state store
> -
>
> Key: YARN-891
> URL: https://issues.apache.org/jira/browse/YARN-891
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch,
> YARN-891.4.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch,
> YARN-891.patch, YARN-891.patch, YARN-891.patch
>
>
> Store completed application/attempt info in RMStateStore when
> application/attempt completes. This solves some problems like finished
> application get lost after RM restart and some other races like YARN-1195
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803561#comment-13803561 ] Bikas Saha commented on YARN-1343: -- We will need to add the update type into the message that goes to the AM. The current code in YARN informs the AM about unhealthy/healthy status of the nodes that the AM knows about. For the reconnect case, the AM already knows about the node and so an additional data field is needed to tell it about reconnection etc. Btw, reading the reconnect code looks like it has a bug because if the node resource changes then it does not change the RMNodeImpl actually stored in the rmContext map. So the node information does not get updated in the RM. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803565#comment-13803565 ] Bikas Saha commented on YARN-1343: -- Also, to be clear, the original code was added to update the AM about unhealthy/healthy node changes in order to reach compatibility with MR1. Here we are making an addition to that mechanism to also pass the added/reconnected nodes. That is a useful improvement. We can also pass removed node information to complete all the cases. My suggestion is to pass the additional information in the NodeStatus object that is sent to the AM. The additional status would be what the change was. addition, unhealthy, healthy, restarted/reconnected, removed. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803568#comment-13803568 ] Alejandro Abdelnur commented on YARN-1343: -- bq. My suggestion is to pass the additional information in the NodeStatus object that is sent to the AM. The additional status would be what the change was. addition, unhealthy, healthy, restarted/reconnected, removed. IMO this suggestion is a separate JIRA and it will required changes in the API. What this JIRA is fixing a bug where a AM never gets notified via updates with a node is RUNNING (for the first time or again), only when a node goes !RUNNING. > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1323) Set HTTPS webapp address along with other RPC addresses
[ https://issues.apache.org/jira/browse/YARN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803574#comment-13803574 ] Tsuyoshi OZAWA commented on YARN-1323: -- +1 > Set HTTPS webapp address along with other RPC addresses > --- > > Key: YARN-1323 > URL: https://issues.apache.org/jira/browse/YARN-1323 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1323-1.patch > > > YARN-1232 adds the ability to configure multiple RMs, but missed out the > https web app address. Need to add that in. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1323) Set HTTPS webapp address along with other RPC addresses
[ https://issues.apache.org/jira/browse/YARN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1323: - Hadoop Flags: Reviewed > Set HTTPS webapp address along with other RPC addresses > --- > > Key: YARN-1323 > URL: https://issues.apache.org/jira/browse/YARN-1323 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1323-1.patch > > > YARN-1232 adds the ability to configure multiple RMs, but missed out the > https web app address. Need to add that in. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-947: - Attachment: YARN-947.8.patch Uploaded the patch: bq. We don't need a new YarnContainerState, already have ContainerState. Rollback to ContainerState. bq. SchedulerUtils.java changes are unnecessary. Cleaned. bq. Now we can only modify the yarn_protos and YarnApplicationAttemptState to add the new state enum. Added. bq. Remove PBImpl, and change HistoryData to a simple object that containing of the fields. Done > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch, YARN-947.8.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[
https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803639#comment-13803639
]
Hadoop QA commented on YARN-947:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609967/YARN-947.8.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2271//console
This message is automatically generated.
> Defining the history data classes for the implementation of the
> reading/writing interface
> -
>
> Key: YARN-947
> URL: https://issues.apache.org/jira/browse/YARN-947
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: YARN-321
>
> Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch,
> YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch, YARN-947.8.patch
>
>
> We need to define the history data classes have the exact fields to be
> stored. Therefore, all the implementations don't need to have the duplicate
> logic to exact the required information from RMApp, RMAppAttempt and
> RMContainer.
> We use protobuf to define these classes, such that they can be ser/des
> to/from bytes, which are easier for persistence.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1068) Add admin support for HA operations
[ https://issues.apache.org/jira/browse/YARN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1068: --- Attachment: yarn-1068-13.patch Added HA-related tests to TestRMAdminCLI. > Add admin support for HA operations > --- > > Key: YARN-1068 > URL: https://issues.apache.org/jira/browse/YARN-1068 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: yarn-1068-10.patch, yarn-1068-11.patch, > yarn-1068-12.patch, yarn-1068-13.patch, yarn-1068-1.patch, yarn-1068-2.patch, > yarn-1068-3.patch, yarn-1068-4.patch, yarn-1068-5.patch, yarn-1068-6.patch, > yarn-1068-7.patch, yarn-1068-8.patch, yarn-1068-9.patch, > yarn-1068-prelim.patch > > > Support HA admin operations to facilitate transitioning the RM to Active and > Standby states. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1335) Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication
[
https://issues.apache.org/jira/browse/YARN-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803642#comment-13803642
]
Tsuyoshi OZAWA commented on YARN-1335:
--
Thank you for taking this JIRA, Sandy. The new class structure looks good to
me. These are my review comments:
FSSchedulerApp#getAllowedLocalityLevel() should check isStopped flag and return
null when isStopped is true.
Comment about FiCaSchedulerApp is same to SchedulerApplication. Why don't we
deleting FiCaSchedulerApp's comment or correct it? If we correct it, comment
about FSSchedulerApp should also be added.
{code}
+
+ public abstract NodeId getNodeID();
+
{code}
It's better to describe the difference between what SchedulerNode#getNodeID()
returns and what SchedulerNode#getNodeName() returns in javadoc of the method.
> Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into
> SchedulerApplication
> --
>
> Key: YARN-1335
> URL: https://issues.apache.org/jira/browse/YARN-1335
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 2.2.0
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-1335.patch
>
>
> FSSchedulerApp and FiCaSchedulerApp use duplicate code in a lot of places.
> They both extend SchedulerApplication. We can move a lot of this duplicate
> code into SchedulerApplication.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1343) NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs
[ https://issues.apache.org/jira/browse/YARN-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803648#comment-13803648 ] Bikas Saha commented on YARN-1343: -- This is not a bug since sending the AM information about added/removed nodes is a feature that was never added to the RM. The feature was added to reporting healthy/unhealthy updates for an existing node. So adding support to notify about additions/removals is an improvement and I like this improvement. I am trying to understand how we can make this improvement useful to the AM that receive this information. If we simply send the nodestatus object to the AM how is the AM expected to make sense of it? Are there any existing fields in nodestatus that tell the AM about addition/reconnection? If not, then is this patch complete wrt the intent of the jira to inform AMs about addition/reconnection and (potentially removal)? > NodeManagers additions/restarts are not reported as node updates in > AllocateResponse responses to AMs > - > > Key: YARN-1343 > URL: https://issues.apache.org/jira/browse/YARN-1343 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.2.1 > > Attachments: YARN-1343.patch > > > If a NodeManager joins the cluster or gets restarted, running AMs never > receive the node update indicating the Node is running. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1068) Add admin support for HA operations
[
https://issues.apache.org/jira/browse/YARN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803662#comment-13803662
]
Hadoop QA commented on YARN-1068:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609970/yarn-1068-13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2272//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2272//console
This message is automatically generated.
> Add admin support for HA operations
> ---
>
> Key: YARN-1068
> URL: https://issues.apache.org/jira/browse/YARN-1068
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: ha
> Attachments: yarn-1068-10.patch, yarn-1068-11.patch,
> yarn-1068-12.patch, yarn-1068-13.patch, yarn-1068-1.patch, yarn-1068-2.patch,
> yarn-1068-3.patch, yarn-1068-4.patch, yarn-1068-5.patch, yarn-1068-6.patch,
> yarn-1068-7.patch, yarn-1068-8.patch, yarn-1068-9.patch,
> yarn-1068-prelim.patch
>
>
> Support HA admin operations to facilitate transitioning the RM to Active and
> Standby states.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-473) Capacity Scheduler webpage and REST API not showing correct number of pending applications
[ https://issues.apache.org/jira/browse/YARN-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803666#comment-13803666 ] Jonathan Eagles commented on YARN-473: -- I haven't seen any updates on this, so assigning this to another contributor. Feel free chime in if you're still wanting this. I'd like to get this committed in the next week or so. > Capacity Scheduler webpage and REST API not showing correct number of pending > applications > -- > > Key: YARN-473 > URL: https://issues.apache.org/jira/browse/YARN-473 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 0.23.6 >Reporter: Kendall Thrapp >Assignee: Timothy Chen > Labels: usability > > The Capacity Scheduler REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API) > is not returning the correct number of pending applications. > numPendingApplications is almost always zero, even if there are dozens of > pending apps. > In investigating this, I discovered that the Resource Manager's Scheduler > webpage is also showing an incorrect but different number of pending > applications. For example, the cluster I'm looking at right now currently > has 15 applications in the ACCEPTED state, but the Cluster Metrics table near > the top of the page says there are only 2 pending apps. The REST API says > there are zero pending apps. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-473) Capacity Scheduler webpage and REST API not showing correct number of pending applications
[ https://issues.apache.org/jira/browse/YARN-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-473: - Assignee: Mit Desai (was: Timothy Chen) > Capacity Scheduler webpage and REST API not showing correct number of pending > applications > -- > > Key: YARN-473 > URL: https://issues.apache.org/jira/browse/YARN-473 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 0.23.6 >Reporter: Kendall Thrapp >Assignee: Mit Desai > Labels: usability > > The Capacity Scheduler REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API) > is not returning the correct number of pending applications. > numPendingApplications is almost always zero, even if there are dozens of > pending apps. > In investigating this, I discovered that the Resource Manager's Scheduler > webpage is also showing an incorrect but different number of pending > applications. For example, the cluster I'm looking at right now currently > has 15 applications in the ACCEPTED state, but the Cluster Metrics table near > the top of the page says there are only 2 pending apps. The REST API says > there are zero pending apps. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1321) NMTokenCache is a a singleton, prevents multiple AMs running in a single JVM to work correctly
[
https://issues.apache.org/jira/browse/YARN-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803703#comment-13803703
]
Vinod Kumar Vavilapalli commented on YARN-1321:
---
I think we should try to keep both the client libraries decoupled from each
other. These are the cases:
- Common use case - using both libraries: User's AM doesn't do anything today.
- Less likely
-- Using AMRMClient only: User's AM access the static token-cache to
extract tokens and use them for talking to NMs.
-- Using NMClient only: User's AM gets tokens from the protocol response
and puts them into the static cache.
Looks like whatever we do, we'll break compat. So, as much as I hate to do it,
we can add a config yarn.client.static-nmtokens-cache that is set to true by
default. If it is explicitly set to false (by llama), then AMRMClient and
NMClient can mandate setting the token-cache via setNMTokenCache() APIs.
- That way, when used as a singleton, all the above use-cases would just work
as they are now.
- Otherwise, when the flag is unset, we go to a mode where all AMs should
create Clients explicitly passing a NMTokenCache for use. Otherwise, client
creation will fail.
Would that work? What do others think? Am I still sane?
> NMTokenCache is a a singleton, prevents multiple AMs running in a single JVM
> to work correctly
> --
>
> Key: YARN-1321
> URL: https://issues.apache.org/jira/browse/YARN-1321
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.2.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>Priority: Blocker
> Attachments: YARN-1321.patch, YARN-1321.patch, YARN-1321.patch,
> YARN-1321.patch
>
>
> NMTokenCache is a singleton. Because of this, if running multiple AMs in a
> single JVM NMTokens for the same node from different AMs step on each other
> and starting containers fail due to mismatch tokens.
> The error observed in the client side is something like:
> {code}
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:llama (auth:PROXY) via llama (auth:SIMPLE)
> cause:org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request
> to start container.
> NMToken for application attempt : appattempt_1382038445650_0002_01 was
> used for starting container with container token issued for application
> attempt : appattempt_1382038445650_0001_01
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-934) HistoryStorage writer interface for Application History Server
[ https://issues.apache.org/jira/browse/YARN-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803708#comment-13803708 ] Vinod Kumar Vavilapalli commented on YARN-934: -- Looks good to me too. Will check it in. > HistoryStorage writer interface for Application History Server > -- > > Key: YARN-934 > URL: https://issues.apache.org/jira/browse/YARN-934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-934.1.patch, YARN-934.2.patch, YARN-934.3.patch, > YARN-934.4.patch, YARN-934.5.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-956) [YARN-321] Add a testable in-memory HistoryStorage
[ https://issues.apache.org/jira/browse/YARN-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803714#comment-13803714 ] Vinod Kumar Vavilapalli commented on YARN-956: -- Memory-store has always been only for testing. There is no use for it outside testing. > [YARN-321] Add a testable in-memory HistoryStorage > --- > > Key: YARN-956 > URL: https://issues.apache.org/jira/browse/YARN-956 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-956-1.patch, YARN-956-2.patch, YARN-956-3.patch, > YARN-956.4.patch, YARN-956.5.patch, YARN-956.6.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803716#comment-13803716 ] Vinod Kumar Vavilapalli commented on YARN-947: -- Looks good, +1. Will check it in. Please file all the follow up tickets. > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch, YARN-947.8.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-891) Store completed application information in RM state store
[ https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-891: - Attachment: YARN-891.5.patch New patch revert RMAppAttemptImpl.finalTrackingUrl back to RMAppAttemptImpl.originalTrackingUrl and fixed a miss in setting up the diagnostics for failed apps > Store completed application information in RM state store > - > > Key: YARN-891 > URL: https://issues.apache.org/jira/browse/YARN-891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch, > YARN-891.4.patch, YARN-891.5.patch, YARN-891.patch, YARN-891.patch, > YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch > > > Store completed application/attempt info in RMStateStore when > application/attempt completes. This solves some problems like finished > application get lost after RM restart and some other races like YARN-1195 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-947) Defining the history data classes for the implementation of the reading/writing interface
[ https://issues.apache.org/jira/browse/YARN-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803743#comment-13803743 ] Zhijie Shen commented on YARN-947: -- Thanks for the review. Will file the follow up tickets. As it's ready to go. I'll update YARN-956 and YARN-975 accordingly > Defining the history data classes for the implementation of the > reading/writing interface > - > > Key: YARN-947 > URL: https://issues.apache.org/jira/browse/YARN-947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-947.1.patch, YARN-947.2.patch, YARN-947.3.patch, > YARN-947.4.patch, YARN-947.5.patch, YARN-947.6.patch, YARN-947.8.patch > > > We need to define the history data classes have the exact fields to be > stored. Therefore, all the implementations don't need to have the duplicate > logic to exact the required information from RMApp, RMAppAttempt and > RMContainer. > We use protobuf to define these classes, such that they can be ser/des > to/from bytes, which are easier for persistence. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-956) [YARN-321] Add a testable in-memory HistoryStorage
[ https://issues.apache.org/jira/browse/YARN-956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-956: - Attachment: YARN-956.7.patch Do the following things in the new patch: 1. Rebase against the new patch in YARN-947 2. Address Vinod's comments 3. I mark the memory implementation \@Private and add some javadoc to inform users of not constructing it. > [YARN-321] Add a testable in-memory HistoryStorage > --- > > Key: YARN-956 > URL: https://issues.apache.org/jira/browse/YARN-956 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Zhijie Shen > Fix For: YARN-321 > > Attachments: YARN-956-1.patch, YARN-956-2.patch, YARN-956-3.patch, > YARN-956.4.patch, YARN-956.5.patch, YARN-956.6.patch, YARN-956.7.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-956) [YARN-321] Add a testable in-memory HistoryStorage
[
https://issues.apache.org/jira/browse/YARN-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803745#comment-13803745
]
Hadoop QA commented on YARN-956:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609988/YARN-956.7.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2273//console
This message is automatically generated.
> [YARN-321] Add a testable in-memory HistoryStorage
> ---
>
> Key: YARN-956
> URL: https://issues.apache.org/jira/browse/YARN-956
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Zhijie Shen
> Fix For: YARN-321
>
> Attachments: YARN-956-1.patch, YARN-956-2.patch, YARN-956-3.patch,
> YARN-956.4.patch, YARN-956.5.patch, YARN-956.6.patch, YARN-956.7.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-891) Store completed application information in RM state store
[
https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803765#comment-13803765
]
Hadoop QA commented on YARN-891:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609987/YARN-891.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2274//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2274//console
This message is automatically generated.
> Store completed application information in RM state store
> -
>
> Key: YARN-891
> URL: https://issues.apache.org/jira/browse/YARN-891
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch,
> YARN-891.4.patch, YARN-891.5.patch, YARN-891.patch, YARN-891.patch,
> YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch
>
>
> Store completed application/attempt info in RMStateStore when
> application/attempt completes. This solves some problems like finished
> application get lost after RM restart and some other races like YARN-1195
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-891) Store completed application information in RM state store
[ https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-891: - Attachment: YARN-891.6.patch Fixed the test failure > Store completed application information in RM state store > - > > Key: YARN-891 > URL: https://issues.apache.org/jira/browse/YARN-891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch, > YARN-891.4.patch, YARN-891.5.patch, YARN-891.6.patch, YARN-891.patch, > YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch > > > Store completed application/attempt info in RMStateStore when > application/attempt completes. This solves some problems like finished > application get lost after RM restart and some other races like YARN-1195 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-891) Store completed application information in RM state store
[
https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803888#comment-13803888
]
Hadoop QA commented on YARN-891:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609997/YARN-891.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2275//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2275//console
This message is automatically generated.
> Store completed application information in RM state store
> -
>
> Key: YARN-891
> URL: https://issues.apache.org/jira/browse/YARN-891
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bikas Saha
>Assignee: Jian He
> Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch,
> YARN-891.4.patch, YARN-891.5.patch, YARN-891.6.patch, YARN-891.patch,
> YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch
>
>
> Store completed application/attempt info in RMStateStore when
> application/attempt completes. This solves some problems like finished
> application get lost after RM restart and some other races like YARN-1195
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1335) Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication
[ https://issues.apache.org/jira/browse/YARN-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1335: - Attachment: YARN-1335-1.patch > Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into > SchedulerApplication > -- > > Key: YARN-1335 > URL: https://issues.apache.org/jira/browse/YARN-1335 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1335-1.patch, YARN-1335.patch > > > FSSchedulerApp and FiCaSchedulerApp use duplicate code in a lot of places. > They both extend SchedulerApplication. We can move a lot of this duplicate > code into SchedulerApplication. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1335) Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication
[ https://issues.apache.org/jira/browse/YARN-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803891#comment-13803891 ] Sandy Ryza commented on YARN-1335: -- Thanks for the review Tsuyoshi. Uploading a patch that adds javadoc for SchedulerNode#getNodename, cleans up javadoc for FSSchedulerApp, FiCaSchedulerApp, and SchedulerApplication. Regarding getAllowedLocalityLevel, I don't think it needs the isStopped guard because it doesn't mutate any state. Let me know if I'm missing anything here. > Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into > SchedulerApplication > -- > > Key: YARN-1335 > URL: https://issues.apache.org/jira/browse/YARN-1335 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1335-1.patch, YARN-1335.patch > > > FSSchedulerApp and FiCaSchedulerApp use duplicate code in a lot of places. > They both extend SchedulerApplication. We can move a lot of this duplicate > code into SchedulerApplication. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-975) Add a file-system implementation for history-storage
[ https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-975: - Attachment: YARN-975.10.patch Update the patch according the change in YARN-947 and YARN-956 > Add a file-system implementation for history-storage > > > Key: YARN-975 > URL: https://issues.apache.org/jira/browse/YARN-975 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-975.10.patch, YARN-975.1.patch, YARN-975.2.patch, > YARN-975.3.patch, YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch, > YARN-975.7.patch, YARN-975.8.patch, YARN-975.9.patch > > > HDFS implementation should be a standard persistence strategy of history > storage -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-975) Add a file-system implementation for history-storage
[
https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803902#comment-13803902
]
Hadoop QA commented on YARN-975:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610063/YARN-975.10.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2277//console
This message is automatically generated.
> Add a file-system implementation for history-storage
>
>
> Key: YARN-975
> URL: https://issues.apache.org/jira/browse/YARN-975
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-975.10.patch, YARN-975.1.patch, YARN-975.2.patch,
> YARN-975.3.patch, YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch,
> YARN-975.7.patch, YARN-975.8.patch, YARN-975.9.patch
>
>
> HDFS implementation should be a standard persistence strategy of history
> storage
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1344) Separate ApplicationAttemptStartDataProto and ApplicationAttemptRegisteredDataProto
[ https://issues.apache.org/jira/browse/YARN-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1344: -- Issue Type: Sub-task (was: Improvement) Parent: YARN-321 > Separate ApplicationAttemptStartDataProto and > ApplicationAttemptRegisteredDataProto > --- > > Key: YARN-1344 > URL: https://issues.apache.org/jira/browse/YARN-1344 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Some info in ApplicationAttemptStartData can separated, and put into > ApplicationAttemptRegisteredData, to further minimize the info loss > probability. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (YARN-1344) Separate ApplicationAttemptStartDataProto and ApplicationAttemptRegisteredDataProto
Zhijie Shen created YARN-1344: - Summary: Separate ApplicationAttemptStartDataProto and ApplicationAttemptRegisteredDataProto Key: YARN-1344 URL: https://issues.apache.org/jira/browse/YARN-1344 Project: Hadoop YARN Issue Type: Improvement Reporter: Zhijie Shen Assignee: Zhijie Shen Some info in ApplicationAttemptStartData can separated, and put into ApplicationAttemptRegisteredData, to further minimize the info loss probability. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-891) Store completed application information in RM state store
[ https://issues.apache.org/jira/browse/YARN-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803906#comment-13803906 ] Jian He commented on YARN-891: -- The TestZKRMStateStoreZKClientConnections failure is unrelated. > Store completed application information in RM state store > - > > Key: YARN-891 > URL: https://issues.apache.org/jira/browse/YARN-891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Jian He > Attachments: YARN-891.1.patch, YARN-891.2.patch, YARN-891.3.patch, > YARN-891.4.patch, YARN-891.5.patch, YARN-891.6.patch, YARN-891.patch, > YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch, YARN-891.patch > > > Store completed application/attempt info in RMStateStore when > application/attempt completes. This solves some problems like finished > application get lost after RM restart and some other races like YARN-1195 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (YARN-1345) RMServerUtils#createApplicationAttemptState needs to add the mapping for FINAL_SAVING
Zhijie Shen created YARN-1345: - Summary: RMServerUtils#createApplicationAttemptState needs to add the mapping for FINAL_SAVING Key: YARN-1345 URL: https://issues.apache.org/jira/browse/YARN-1345 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Whenever YARN-891 is done, we need to add the mapping of RMAppAttemptState.FINAL_SAVING -> YarnApplicationAttemptState.FINAL_SAVING in RMServerUtils#createApplicationAttemptState -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1335) Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication
[
https://issues.apache.org/jira/browse/YARN-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803915#comment-13803915
]
Hadoop QA commented on YARN-1335:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610061/YARN-1335-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2276//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2276//console
This message is automatically generated.
> Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into
> SchedulerApplication
> --
>
> Key: YARN-1335
> URL: https://issues.apache.org/jira/browse/YARN-1335
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 2.2.0
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-1335-1.patch, YARN-1335.patch
>
>
> FSSchedulerApp and FiCaSchedulerApp use duplicate code in a lot of places.
> They both extend SchedulerApplication. We can move a lot of this duplicate
> code into SchedulerApplication.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (YARN-1346) Revisit the output type of the reader interface
Zhijie Shen created YARN-1346: - Summary: Revisit the output type of the reader interface Key: YARN-1346 URL: https://issues.apache.org/jira/browse/YARN-1346 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen In YARN-947, there's a discussion in YARN-947 about changing the reader interface to return the report protobuf (e.g., ApplicationReport) directly instead of AHS internal objects (e.g., ApplicationHistoryData). We need to think more about it. -- This message was sent by Atlassian JIRA (v6.1#6144)
