[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309301#comment-14309301 ] Hudson commented on YARN-1582: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2047 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2047/]) YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves (jlowe: rev 69c8a7f45be5c0aa6787b07f328d74f1e2ba5628) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.7.0 > > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309302#comment-14309302
]
Hudson commented on YARN-1537:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2047 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2047/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
* hadoop-yarn-project/CHANGES.txt
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2246) Job History Link in RM UI is redirecting to the URL which contains Job Id twice
[
https://issues.apache.org/jira/browse/YARN-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309280#comment-14309280
]
Jason Lowe commented on YARN-2246:
--
[~devaraj.k] are you still planning to address this issue? It's a benign
problem with the history server UI since it ignores the extra components of the
URL, but there are some use cases with Tez and other instances where this needs
to be fixed.
> Job History Link in RM UI is redirecting to the URL which contains Job Id
> twice
> ---
>
> Key: YARN-2246
> URL: https://issues.apache.org/jira/browse/YARN-2246
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 3.0.0, 0.23.11, 2.5.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-4064-1.patch, MAPREDUCE-4064.patch
>
>
> {code:xml}
> http://xx.x.x.x:19888/jobhistory/job/job_1332435449546_0001/jobhistory/job/job_1332435449546_0001
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2809) Implement workaround for linux kernel panic when removing cgroup
[ https://issues.apache.org/jira/browse/YARN-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-2809: - Attachment: YARN-2809-v2.patch upmerge to latest trunk > Implement workaround for linux kernel panic when removing cgroup > > > Key: YARN-2809 > URL: https://issues.apache.org/jira/browse/YARN-2809 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: RHEL 6.4 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Attachments: YARN-2809-v2.patch, YARN-2809.patch > > > Some older versions of linux have a bug that can cause a kernel panic when > the LCE attempts to remove a cgroup. It is a race condition so it's a bit > rare but on a few thousand node cluster it can result in a couple of panics > per day. > This is the commit that likely (haven't verified) fixes the problem in linux: > https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-2.6.39.y&id=068c5cc5ac7414a8e9eb7856b4bf3cc4d4744267 > Details will be added in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3145) ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo
[
https://issues.apache.org/jira/browse/YARN-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309258#comment-14309258
]
Hudson commented on YARN-3145:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/])
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler
ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev
4641196fe02af5cab3d56a9f3c78875c495dbe03)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
> ConcurrentModificationException on CapacityScheduler
> ParentQueue#getQueueUserAclInfo
>
>
> Key: YARN-3145
> URL: https://issues.apache.org/jira/browse/YARN-3145
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-3145.001.patch, YARN-3145.002.patch
>
>
> {code}
> ava.util.ConcurrentModificationException(java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:348)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueUserAclInfo(CapacityScheduler.java:850)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:844)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueUserAcls(ApplicationClientProtocolPBServiceImpl.java:250)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:335)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309262#comment-14309262 ] Hudson commented on YARN-3149: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/]) YARN-3149. Fix typo in message for invalid application id. Contributed (xgong: rev b77ff37686e01b7497d3869fbc62789a5b123c0a) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java > Typo in message for invalid application id > -- > > Key: YARN-3149 > URL: https://issues.apache.org/jira/browse/YARN-3149 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.7.0 > > Attachments: YARN-3149.patch, YARN-3149.patch, screenshot-1.png > > > Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1904) Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService
[ https://issues.apache.org/jira/browse/YARN-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309259#comment-14309259 ] Hudson commented on YARN-1904: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/]) YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen. (acmurthy: rev 18b2507edaac991e3ed68d2f27eb96f6882137b9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java * hadoop-yarn-project/CHANGES.txt > Uniform the NotFound messages from ClientRMService and > ApplicationHistoryClientService > -- > > Key: YARN-1904 > URL: https://issues.apache.org/jira/browse/YARN-1904 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.7.0 > > Attachments: YARN-1904.1.patch > > > It's good to make ClientRMService and ApplicationHistoryClientService throw > NotFoundException with similar messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) In Fair Scheduler, fix canceling of reservations for exceeding max share
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309255#comment-14309255 ] Hudson commented on YARN-3101: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/]) YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza) (sandy: rev b6466deac6d5d6344f693144290b46e2bef83a02) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/CHANGES.txt > In Fair Scheduler, fix canceling of reservations for exceeding max share > > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Fix For: 2.7.0 > > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309252#comment-14309252 ] Hudson commented on YARN-1582: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/]) YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves (jlowe: rev 69c8a7f45be5c0aa6787b07f328d74f1e2ba5628) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.7.0 > > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309253#comment-14309253
]
Hudson commented on YARN-1537:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) In Fair Scheduler, fix canceling of reservations for exceeding max share
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309199#comment-14309199 ] Hudson commented on YARN-3101: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/]) YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza) (sandy: rev b6466deac6d5d6344f693144290b46e2bef83a02) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java > In Fair Scheduler, fix canceling of reservations for exceeding max share > > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Fix For: 2.7.0 > > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1904) Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService
[ https://issues.apache.org/jira/browse/YARN-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309203#comment-14309203 ] Hudson commented on YARN-1904: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/]) YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen. (acmurthy: rev 18b2507edaac991e3ed68d2f27eb96f6882137b9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/CHANGES.txt > Uniform the NotFound messages from ClientRMService and > ApplicationHistoryClientService > -- > > Key: YARN-1904 > URL: https://issues.apache.org/jira/browse/YARN-1904 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.7.0 > > Attachments: YARN-1904.1.patch > > > It's good to make ClientRMService and ApplicationHistoryClientService throw > NotFoundException with similar messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309196#comment-14309196
]
Hudson commented on YARN-1537:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
* hadoop-yarn-project/CHANGES.txt
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309206#comment-14309206 ] Hudson commented on YARN-3149: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/]) YARN-3149. Fix typo in message for invalid application id. Contributed (xgong: rev b77ff37686e01b7497d3869fbc62789a5b123c0a) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java > Typo in message for invalid application id > -- > > Key: YARN-3149 > URL: https://issues.apache.org/jira/browse/YARN-3149 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.7.0 > > Attachments: YARN-3149.patch, YARN-3149.patch, screenshot-1.png > > > Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3145) ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo
[
https://issues.apache.org/jira/browse/YARN-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309202#comment-14309202
]
Hudson commented on YARN-3145:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/])
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler
ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev
4641196fe02af5cab3d56a9f3c78875c495dbe03)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
> ConcurrentModificationException on CapacityScheduler
> ParentQueue#getQueueUserAclInfo
>
>
> Key: YARN-3145
> URL: https://issues.apache.org/jira/browse/YARN-3145
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-3145.001.patch, YARN-3145.002.patch
>
>
> {code}
> ava.util.ConcurrentModificationException(java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:348)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueUserAclInfo(CapacityScheduler.java:850)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:844)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueUserAcls(ApplicationClientProtocolPBServiceImpl.java:250)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:335)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309195#comment-14309195 ] Hudson commented on YARN-1582: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/]) YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves (jlowe: rev 69c8a7f45be5c0aa6787b07f328d74f1e2ba5628) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.7.0 > > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309170#comment-14309170 ] Hudson commented on YARN-1582: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/]) YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves (jlowe: rev 69c8a7f45be5c0aa6787b07f328d74f1e2ba5628) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.7.0 > > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3145) ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo
[
https://issues.apache.org/jira/browse/YARN-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309176#comment-14309176
]
Hudson commented on YARN-3145:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/])
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler
ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev
4641196fe02af5cab3d56a9f3c78875c495dbe03)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
> ConcurrentModificationException on CapacityScheduler
> ParentQueue#getQueueUserAclInfo
>
>
> Key: YARN-3145
> URL: https://issues.apache.org/jira/browse/YARN-3145
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-3145.001.patch, YARN-3145.002.patch
>
>
> {code}
> ava.util.ConcurrentModificationException(java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:348)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueUserAclInfo(CapacityScheduler.java:850)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:844)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueUserAcls(ApplicationClientProtocolPBServiceImpl.java:250)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:335)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309180#comment-14309180 ] Hudson commented on YARN-3149: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/]) YARN-3149. Fix typo in message for invalid application id. Contributed (xgong: rev b77ff37686e01b7497d3869fbc62789a5b123c0a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java * hadoop-yarn-project/CHANGES.txt > Typo in message for invalid application id > -- > > Key: YARN-3149 > URL: https://issues.apache.org/jira/browse/YARN-3149 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.7.0 > > Attachments: YARN-3149.patch, YARN-3149.patch, screenshot-1.png > > > Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309171#comment-14309171
]
Hudson commented on YARN-1537:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
* hadoop-yarn-project/CHANGES.txt
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) In Fair Scheduler, fix canceling of reservations for exceeding max share
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309173#comment-14309173 ] Hudson commented on YARN-3101: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/]) YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza) (sandy: rev b6466deac6d5d6344f693144290b46e2bef83a02) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java > In Fair Scheduler, fix canceling of reservations for exceeding max share > > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Fix For: 2.7.0 > > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1904) Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService
[ https://issues.apache.org/jira/browse/YARN-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309177#comment-14309177 ] Hudson commented on YARN-1904: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/]) YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen. (acmurthy: rev 18b2507edaac991e3ed68d2f27eb96f6882137b9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java * hadoop-yarn-project/CHANGES.txt > Uniform the NotFound messages from ClientRMService and > ApplicationHistoryClientService > -- > > Key: YARN-1904 > URL: https://issues.apache.org/jira/browse/YARN-1904 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.7.0 > > Attachments: YARN-1904.1.patch > > > It's good to make ClientRMService and ApplicationHistoryClientService throw > NotFoundException with similar messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1904) Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService
[ https://issues.apache.org/jira/browse/YARN-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309014#comment-14309014 ] Hudson commented on YARN-1904: -- FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/830/]) YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen. (acmurthy: rev 18b2507edaac991e3ed68d2f27eb96f6882137b9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java * hadoop-yarn-project/CHANGES.txt > Uniform the NotFound messages from ClientRMService and > ApplicationHistoryClientService > -- > > Key: YARN-1904 > URL: https://issues.apache.org/jira/browse/YARN-1904 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.7.0 > > Attachments: YARN-1904.1.patch > > > It's good to make ClientRMService and ApplicationHistoryClientService throw > NotFoundException with similar messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309018#comment-14309018 ] Hudson commented on YARN-3149: -- FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/830/]) YARN-3149. Fix typo in message for invalid application id. Contributed (xgong: rev b77ff37686e01b7497d3869fbc62789a5b123c0a) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java > Typo in message for invalid application id > -- > > Key: YARN-3149 > URL: https://issues.apache.org/jira/browse/YARN-3149 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.7.0 > > Attachments: YARN-3149.patch, YARN-3149.patch, screenshot-1.png > > > Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3145) ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo
[
https://issues.apache.org/jira/browse/YARN-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309013#comment-14309013
]
Hudson commented on YARN-3145:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/830/])
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler
ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev
4641196fe02af5cab3d56a9f3c78875c495dbe03)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
> ConcurrentModificationException on CapacityScheduler
> ParentQueue#getQueueUserAclInfo
>
>
> Key: YARN-3145
> URL: https://issues.apache.org/jira/browse/YARN-3145
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-3145.001.patch, YARN-3145.002.patch
>
>
> {code}
> ava.util.ConcurrentModificationException(java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:348)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueUserAclInfo(CapacityScheduler.java:850)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:844)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueUserAcls(ApplicationClientProtocolPBServiceImpl.java:250)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:335)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) In Fair Scheduler, fix canceling of reservations for exceeding max share
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309010#comment-14309010 ] Hudson commented on YARN-3101: -- FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/830/]) YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza) (sandy: rev b6466deac6d5d6344f693144290b46e2bef83a02) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java > In Fair Scheduler, fix canceling of reservations for exceeding max share > > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Fix For: 2.7.0 > > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309007#comment-14309007 ] Hudson commented on YARN-1582: -- FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/830/]) YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves (jlowe: rev 69c8a7f45be5c0aa6787b07f328d74f1e2ba5628) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.7.0 > > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309008#comment-14309008
]
Hudson commented on YARN-1537:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/830/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3100) Make YARN authorization pluggable
[
https://issues.apache.org/jira/browse/YARN-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308987#comment-14308987
]
Chris Douglas commented on YARN-3100:
-
Looking through {{AbstractCSQueue}} and {{CSQueueUtils}}, it looks like there
are many misconfigurations that leave queues in an inconsistent state...
> Make YARN authorization pluggable
> -
>
> Key: YARN-3100
> URL: https://issues.apache.org/jira/browse/YARN-3100
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-3100.1.patch, YARN-3100.2.patch
>
>
> The goal is to have YARN acl model pluggable so as to integrate other
> authorization tool such as Apache Ranger, Sentry.
> Currently, we have
> - admin ACL
> - queue ACL
> - application ACL
> - time line domain ACL
> - service ACL
> The proposal is to create a YarnAuthorizationProvider interface. Current
> implementation will be the default implementation. Ranger or Sentry plug-in
> can implement this interface.
> Benefit:
> - Unify the code base. With the default implementation, we can get rid of
> each specific ACL manager such as AdminAclManager, ApplicationACLsManager,
> QueueAclsManager etc.
> - Enable Ranger, Sentry to do authorization for YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3145) ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo
[
https://issues.apache.org/jira/browse/YARN-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308952#comment-14308952
]
Hudson commented on YARN-3145:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/])
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler
ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev
4641196fe02af5cab3d56a9f3c78875c495dbe03)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* hadoop-yarn-project/CHANGES.txt
> ConcurrentModificationException on CapacityScheduler
> ParentQueue#getQueueUserAclInfo
>
>
> Key: YARN-3145
> URL: https://issues.apache.org/jira/browse/YARN-3145
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Tsuyoshi OZAWA
> Fix For: 2.7.0
>
> Attachments: YARN-3145.001.patch, YARN-3145.002.patch
>
>
> {code}
> ava.util.ConcurrentModificationException(java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:348)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueUserAclInfo(CapacityScheduler.java:850)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:844)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueUserAcls(ApplicationClientProtocolPBServiceImpl.java:250)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:335)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308957#comment-14308957 ] Hudson commented on YARN-3149: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/]) YARN-3149. Fix typo in message for invalid application id. Contributed (xgong: rev b77ff37686e01b7497d3869fbc62789a5b123c0a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java * hadoop-yarn-project/CHANGES.txt > Typo in message for invalid application id > -- > > Key: YARN-3149 > URL: https://issues.apache.org/jira/browse/YARN-3149 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Trivial > Fix For: 2.7.0 > > Attachments: YARN-3149.patch, YARN-3149.patch, screenshot-1.png > > > Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308946#comment-14308946 ] Hudson commented on YARN-1582: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/]) YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves (jlowe: rev 69c8a7f45be5c0aa6787b07f328d74f1e2ba5628) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.7.0 > > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1904) Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService
[ https://issues.apache.org/jira/browse/YARN-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308953#comment-14308953 ] Hudson commented on YARN-1904: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/]) YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen. (acmurthy: rev 18b2507edaac991e3ed68d2f27eb96f6882137b9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java > Uniform the NotFound messages from ClientRMService and > ApplicationHistoryClientService > -- > > Key: YARN-1904 > URL: https://issues.apache.org/jira/browse/YARN-1904 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.7.0 > > Attachments: YARN-1904.1.patch > > > It's good to make ClientRMService and ApplicationHistoryClientService throw > NotFoundException with similar messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308947#comment-14308947
]
Hudson commented on YARN-1537:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) In Fair Scheduler, fix canceling of reservations for exceeding max share
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308949#comment-14308949 ] Hudson commented on YARN-3101: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/]) YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza) (sandy: rev b6466deac6d5d6344f693144290b46e2bef83a02) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/CHANGES.txt > In Fair Scheduler, fix canceling of reservations for exceeding max share > > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Fix For: 2.7.0 > > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-933) After an AppAttempt_1 got failed [ removal and releasing of container is done , AppAttempt_2 is scheduled ] again relaunching of AppAttempt_1 throws Exception at RM .And
[ https://issues.apache.org/jira/browse/YARN-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308862#comment-14308862 ] Rohith commented on YARN-933: - Sure, I will recheck the code for existence of problem and update the patch. > After an AppAttempt_1 got failed [ removal and releasing of container is done > , AppAttempt_2 is scheduled ] again relaunching of AppAttempt_1 throws > Exception at RM .And client exited before appattempt retries got over > -- > > Key: YARN-933 > URL: https://issues.apache.org/jira/browse/YARN-933 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.0.5-alpha >Reporter: J.Andreina >Assignee: Rohith > Attachments: YARN-933.patch > > > am max retries configured as 3 at client and RM side. > Step 1: Install cluster with NM on 2 Machines > Step 2: Make Ping using ip from RM machine to NM1 machine as successful ,But > using Hostname should fail > Step 3: Execute a job > Step 4: After AM [ AppAttempt_1 ] allocation to NM1 machine is done , > connection loss happened. > Observation : > == > After AppAttempt_1 has moved to failed state ,release of container for > AppAttempt_1 and Application removal are successful. New AppAttempt_2 is > sponed. > 1. Then again retry for AppAttempt_1 happens. > 2. Again RM side it is trying to launch AppAttempt_1, hence fails with > InvalidStateTransitonException > 3. Client got exited after AppAttempt_1 is been finished [But actually job is > still running ], while the appattempts configured is 3 and rest appattempts > are all sponed and running. > RMLogs: > == > 2013-07-17 16:22:51,013 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1373952096466_0056_01 State change from SCHEDULED to ALLOCATED > 2013-07-17 16:35:48,171 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: host-10-18-40-15/10.18.40.59:8048. Already tried 36 time(s); > maxRetries=45 > 2013-07-17 16:36:07,091 INFO > org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: > Expired:container_1373952096466_0056_01_01 Timed out after 600 secs > 2013-07-17 16:36:07,093 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_1373952096466_0056_01_01 Container Transitioned from ACQUIRED > to EXPIRED > 2013-07-17 16:36:07,093 INFO > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Registering appattempt_1373952096466_0056_02 > 2013-07-17 16:36:07,131 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application appattempt_1373952096466_0056_01 is done. finalState=FAILED > 2013-07-17 16:36:07,131 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Application removed - appId: application_1373952096466_0056 user: Rex > leaf-queue of parent: root #applications: 35 > 2013-07-17 16:36:07,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application Submission: appattempt_1373952096466_0056_02, > 2013-07-17 16:36:07,138 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1373952096466_0056_02 State change from SUBMITTED to SCHEDULED > 2013-07-17 16:36:30,179 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: host-10-18-40-15/10.18.40.59:8048. Already tried 38 time(s); > maxRetries=45 > 2013-07-17 16:38:36,203 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: host-10-18-40-15/10.18.40.59:8048. Already tried 44 time(s); > maxRetries=45 > 2013-07-17 16:38:56,207 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Error > launching appattempt_1373952096466_0056_01. Got exception: > java.lang.reflect.UndeclaredThrowableException > 2013-07-17 16:38:56,207 ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > LAUNCH_FAILED at FAILED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:630) > at > org.apache.hadoop.yarn.server.
[jira] [Assigned] (YARN-3151) On Failover tracking url wrong in application cli for KILLED application
[
https://issues.apache.org/jira/browse/YARN-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith reassigned YARN-3151:
Assignee: Rohith
> On Failover tracking url wrong in application cli for KILLED application
>
>
> Key: YARN-3151
> URL: https://issues.apache.org/jira/browse/YARN-3151
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, resourcemanager
>Affects Versions: 2.6.0
> Environment: 2 RM HA
>Reporter: Bibin A Chundatt
>Assignee: Rohith
>Priority: Minor
>
> Run an application and kill the same after starting
> Check {color:red} ./yarn application -list -appStates KILLED {color}
> (empty line)
> {quote}
> Application-Id Tracking-URL
> application_1423219262738_0001
> http://:PORT>/cluster/app/application_1423219262738_0001
> {quote}
> Shutdown the active RM1
> Check the same command {color:red} ./yarn application -list -appStates KILLED
> {color} after RM2 is active
> {quote}
> Application-Id Tracking-URL
> application_1423219262738_0001 null
> {quote}
> Tracking url for application is shown as null
> Expected : Same url before failover should be shown
> ApplicationReport .getOriginalTrackingUrl() is null after failover
> org.apache.hadoop.yarn.client.cli.ApplicationCLI
> listApplications(Set appTypes,
> EnumSet appStates)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3151) On Failover tracking url wrong in application cli for KILLED application
Bibin A Chundatt created YARN-3151:
--
Summary: On Failover tracking url wrong in application cli for
KILLED application
Key: YARN-3151
URL: https://issues.apache.org/jira/browse/YARN-3151
Project: Hadoop YARN
Issue Type: Bug
Components: client, resourcemanager
Affects Versions: 2.6.0
Environment: 2 RM HA
Reporter: Bibin A Chundatt
Priority: Minor
Run an application and kill the same after starting
Check {color:red} ./yarn application -list -appStates KILLED {color}
(empty line)
{quote}
Application-Id Tracking-URL
application_1423219262738_0001
http://:PORT>/cluster/app/application_1423219262738_0001
{quote}
Shutdown the active RM1
Check the same command {color:red} ./yarn application -list -appStates KILLED
{color} after RM2 is active
{quote}
Application-Id Tracking-URL
application_1423219262738_0001 null
{quote}
Tracking url for application is shown as null
Expected : Same url before failover should be shown
ApplicationReport .getOriginalTrackingUrl() is null after failover
org.apache.hadoop.yarn.client.cli.ApplicationCLI
listApplications(Set appTypes,
EnumSet appStates)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1537) TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
[
https://issues.apache.org/jira/browse/YARN-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308817#comment-14308817
]
Hudson commented on YARN-1537:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7038 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7038/])
YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong.
(acmurthy: rev 02f154a0016b7321bbe5b09f2da44a9b33797c36)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
* hadoop-yarn-project/CHANGES.txt
> TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
> -
>
> Key: YARN-1537
> URL: https://issues.apache.org/jira/browse/YARN-1537
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Hong Shen
>Assignee: Xuan Gong
> Fix For: 2.7.0
>
> Attachments: YARN-1537.1.patch
>
>
> Here is the error log
> {code}
> Results :
> Failed tests:
> TestLocalResourcesTrackerImpl.testLocalResourceCache:351
> Wanted but not invoked:
> eventHandler.handle(
>
> isA(org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResourceLocalizedEvent)
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestLocalResourcesTrackerImpl.testLocalResourceCache(TestLocalResourcesTrackerImpl.java:351)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
