[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882125#comment-15882125 ] Arun Suresh commented on YARN-5959: --- The testcase failures are un-related. Filed YARN-6231 to track the TestFairScheduler failure. Will be checking this in shortly > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5959.005.patch, YARN-5959-branch-2.001.patch, > YARN-5959-branch-2.002.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch, > YARN-5959-YARN-5085.001.patch, YARN-5959-YARN-5085.002.patch, > YARN-5959-YARN-5085.003.patch, YARN-5959-YARN-5085.004.patch, > YARN-5959-YARN-5085.005.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar to YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882124#comment-15882124
]
Sunil G commented on YARN-5956:
---
Thanks [~lewuathe] for updating the patch. We are almost there.
A major comment, In getApplicationAttemptReport, getApplicationAttempts,
getContainerReport and getContaines, below code is added.
{code}
UserGroupInformation callerUGI = getCallerUgi(appId,
AuditConstants.GET_CONTAINER_REPORT);
RMApp application = verifyUserAccessForRMApp(appId, callerUGI,
AuditConstants.GET_CONTAINER_REPORT,
ApplicationAccessType.VIEW_APP);
boolean allowAccess = checkAccess(callerUGI, application.getUser(),
ApplicationAccessType.VIEW_APP, application);
{code}
Here *verifyUserAccessForRMApp* is already checking {{checkAccess}} for
*VIEW_APP* internally. SO we can avoid duplicate check in all these getters.
Then some more cleanup could be done as *allowAccess* will be true after the
new *verifyUserAccessForRMApp* call. Do you agree?
> Refactor ClientRMService
>
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch,
> YARN-5956.03.patch, YARN-5956.04.patch, YARN-5956.05.patch,
> YARN-5956.06.patch, YARN-5956.07.patch, YARN-5956.08.patch,
> YARN-5956.09.patch, YARN-5956.10.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882114#comment-15882114
]
Sunil G commented on YARN-6207:
---
On an other note, many of these cases are also applicable for FairScheduler
too. Is it planned in same jira or a new one?
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6231) TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on branch-2
[ https://issues.apache.org/jira/browse/YARN-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882106#comment-15882106 ] Arun Suresh commented on YARN-6231: --- [~kasha], looks like something recent broke this > TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on > branch-2 > --- > > Key: YARN-6231 > URL: https://issues.apache.org/jira/browse/YARN-6231 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Arun Suresh >Assignee: Karthik Kambatla > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6231) TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on branch-2
Arun Suresh created YARN-6231: - Summary: TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on branch-2 Key: YARN-6231 URL: https://issues.apache.org/jira/browse/YARN-6231 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.9.0 Reporter: Arun Suresh Assignee: Karthik Kambatla -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882074#comment-15882074
]
Rohith Sharma K S commented on YARN-6027:
-
bq. Is there a reason not to simply use it?
The row key encoding/decoding does not represent properly to user in string
format. For example, flow row key is
*cl%0$uster!���!��!%0$user!dummy_%0$flow%0$*. This decoding again will be an
issue.
# Bytes.toBytes(System.currentTimeMillis()) - converts to bytes, but when
changed to string, the format is unknown
# new String(Bytes.toBytes(System.currentTimeMillis()) OR
Bytes.toString(Bytes.toBytes(System.currentTimeMillis())) results it junk
characters *Zn�^�*
It is better to use as same as UID encoder. May be we can define new interface
as Varun suggested something like
{code}String encodeAsString(T key);
T decode(String[] bytes);
{code}
> Support fromid(offset) filter for /flows API
>
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch,
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch,
> YARN-6027-YARN-5355.0004.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar
> filter for flows/flowRun apps and flow run and flow as well.
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882038#comment-15882038
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 30s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Null passed for non-null parameter of addYieldedReservations(List, Plan,
ReservationId, boolean) in
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PriorityReservationAgent.updateReservation(ReservationId,
String, Plan, ReservationDefinition) Method invoked at
PriorityReservationAgent.java:of addYieldedReservations(List, Plan,
ReservationId, b
[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882015#comment-15882015
]
Hadoop QA commented on YARN-5959:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 5s{color} | {color:orange} root: The patch generated 34 new + 1600
unchanged - 18 fixed = 1634 total (was 1618) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK
v1.8.0_121. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} hadoop-yarn-server-common in the patch passed with
JDK v1.8.0_121. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121
with JDK v1.8.0_121 generated 0 new + 907 unchanged - 1 fixed = 907 total (was
908) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} hadoop-yarn-client in the patch passed with JDK
v1.8.0_121. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{colo
[jira] [Commented] (YARN-6109) Add an ability to convert ChildQueue to ParentQueue
[ https://issues.apache.org/jira/browse/YARN-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882003#comment-15882003 ] Subru Krishnan commented on YARN-6109: -- [~leftnoteasy], I think it should be fairly straightforward to add the check (and cannibalize corresponding tests from existing ones), so it would be great if we can do it inline. If it turns out to be non-trivial work, we can move to a separate JIRA. > Add an ability to convert ChildQueue to ParentQueue > --- > > Key: YARN-6109 > URL: https://issues.apache.org/jira/browse/YARN-6109 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6109.1.patch, YARN-6109.2.patch, > YARN-6109.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5410) Bootstrap Router module
[
https://issues.apache.org/jira/browse/YARN-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882000#comment-15882000
]
Subru Krishnan commented on YARN-5410:
--
Thanks [~leftnoteasy] for your feedback.
bq. I think I understand the purpose of Router. However, since it implements
ClientRMService, it behaves as gateway of RM. I'm not sure if any logics could
be shared with resource-manager module.
bq. If you're confident with router will be independent from RM, I'm fine with
creating this sub module
The Router will only be a *proxy* to the RM and we will be following an
interceptor pattern (ala servlet filters) exactly like we did with the
{{AMRMProxy}} for the AM - RM protocol in YARN-2884. So the Router will depend
on the {{ApplicationClientProtocol}} and {{FederationStateStore}}
(yarn-server-common module) which are independent of RM.
Hope this makes sense?
> Bootstrap Router module
> ---
>
> Key: YARN-5410
> URL: https://issues.apache.org/jira/browse/YARN-5410
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-5410-YARN-2915-v1.patch,
> YARN-5410-YARN-2915-v2.patch, YARN-5410-YARN-2915-v3.patch,
> YARN-5410-YARN-2915-v4.patch, YARN-5410-YARN-2915-v5.patch,
> YARN-5410-YARN-2915-v6.patch
>
>
> As detailed in the proposal in the umbrella JIRA, we are introducing a new
> component that routes client request to appropriate ResourceManager(s). This
> JIRA tracks the creation of a new sub-module for the Router.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881960#comment-15881960 ] Rohith Sharma K S commented on YARN-6027: - bq. I am a little curious about this refactoring. Is the idea to split the current reader unit tests based on this abstract class? Yes, it is refactoring of TestTimelineReaderWebServicesHBaseStorage. This class has everything mixed up which can be moved to separate class which can be re used. bq. it might be better to place it in the reader package and indicate that in the class name also; e.g. AbstractTimelineReaderHBaseTestBase Agree > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch, > YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, > YARN-6027-YARN-5355.0004.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881940#comment-15881940
]
Sunil G commented on YARN-6207:
---
Yes.. There were many corner cases we have discussed in this Jira. I feel we
should have more test cases here to ensure all cases are handled well. Thanks
for considering same.
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881935#comment-15881935
]
Bibin A Chundatt edited comment on YARN-6207 at 2/24/17 4:40 AM:
-
Yes [~sunilg] i meant the same .Will upload the patch with more testcase to
cover all these scenarios too.
was (Author: bibinchundatt):
Yes [~sunilg] i meant the same .
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881935#comment-15881935
]
Bibin A Chundatt commented on YARN-6207:
Yes [~sunilg] i meant the same .
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4122) Add support for GPU as a resource
[ https://issues.apache.org/jira/browse/YARN-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881926#comment-15881926 ] Vinod Kumar Vavilapalli commented on YARN-4122: --- There is renewed activity in this area - see YARN-6223 umbrella and YARN-3926 which supports more resources in a generic manner. > Add support for GPU as a resource > - > > Key: YARN-4122 > URL: https://issues.apache.org/jira/browse/YARN-4122 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: GPUAsAResourceDesign.pdf > > > Use [cgroups > devcies|https://www.kernel.org/doc/Documentation/cgroups/devices.txt] to > isolate GPUs for containers. For docker containers, we could use 'docker run > --device=...'. > Reference: [SLURM Resources isolation through > cgroups|http://slurm.schedmd.com/slurm_ug_2011/SLURM_UserGroup2011_cgroups.pdf]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v006.patch YARN-5385.v006.patch attempts to fix the findbugs errors by wrapping the methods being locked, with a try finally block. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, > YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5517) Add GPU as a resource type for scheduling
[ https://issues.apache.org/jira/browse/YARN-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881918#comment-15881918 ] Vinod Kumar Vavilapalli commented on YARN-5517: --- bq. There is the issue(YARN-3926) which proposed to extend the YARN resource model. bq. But these issues didn’t release yet so I just added GPU resource type like memory and cpu. YARN-3926 is much further along now, though it did take a bit of time. Further, as the size of the patch here demonstrates, it's better to support arbitrary resources in a pluggable manner instead of continuing to hard-code them into the core. I propose we close this as a dup of YARN-3926. > Add GPU as a resource type for scheduling > - > > Key: YARN-5517 > URL: https://issues.apache.org/jira/browse/YARN-5517 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Jaeboo Jeong > Attachments: aggregate_resource_allocation.jpg, > container_example.jpg, RM-scheduler_metrics.jpg, YARN-5517-branch-2.7.1.patch > > > Currently YARN only support scheduling based on memory and cpu. > There is the issue(YARN-3926) which proposed to extend the YARN resource > model. > And there is the issue(YARN-4122) to add support for GPU as a resource using > docker. > But these issues didn’t release yet so I just added GPU resource type like > memory and cpu. > I don’t consider GPU isolation like YARN-4122. > The properties for GPU resource type is similar to cpu core. > mapred-default.xml > mapreduce.map.gpu.cores (default 0) > mapreduce.reduce.gpu.cores(default 0) > yarn.app.mapreduce.am.resource.gpu-cores (default 0) > yarn-default.xml > yarn.scheduler.minimum-allocation-gcores (default 0) > yarn.scheduler.maximum-allocation-gcores (default 8) > yarn.nodemanager.resource.gcores (default 0) > I attached the patch for branch-2.7.1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6223) [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation on YARN
[ https://issues.apache.org/jira/browse/YARN-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881907#comment-15881907 ] Vinod Kumar Vavilapalli commented on YARN-6223: --- bq. I had a clarification regarding GPU scheduling - are you planning to use YARN-3926? bq. 1. I feel the alternate approaches (YARN-4122/YARN-5517) are neither generic nor scaleable solutions. bq. YARN-4122/YARN-5517 are all adding a new GPU resource type to Resource protocol instead of leveraging YARN-3926. Agreed, we should stop hard-cording more resources into the core and instead add them 'on the surface' in a pluggable manner. > [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation > on YARN > > > Key: YARN-6223 > URL: https://issues.apache.org/jira/browse/YARN-6223 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > > As varieties of workloads are moving to YARN, including machine learning / > deep learning which can speed up by leveraging GPU computation power. > Workloads should be able to request GPU from YARN as simple as CPU and memory. > *To make a complete GPU story, we should support following pieces:* > 1) GPU discovery/configuration: Admin can either config GPU resources and > architectures on each node, or more advanced, NodeManager can automatically > discover GPU resources and architectures and report to ResourceManager > 2) GPU scheduling: YARN scheduler should account GPU as a resource type just > like CPU and memory. > 3) GPU isolation/monitoring: once launch a task with GPU resources, > NodeManager should properly isolate and monitor task's resource usage. > For #2, YARN-3926 can support it natively. For #3, YARN-3611 has introduced > an extensible framework to support isolation for different resource types and > different runtimes. > *Related JIRAs:* > There're a couple of JIRAs (YARN-4122/YARN-5517) filed with similar goals but > different solutions: > For scheduling: > - YARN-4122/YARN-5517 are all adding a new GPU resource type to Resource > protocol instead of leveraging YARN-3926. > For isolation: > - And YARN-4122 proposed to use CGroups to do isolation which cannot solve > the problem listed at > https://github.com/NVIDIA/nvidia-docker/wiki/GPU-isolation#challenges such as > minor device number mapping; load nvidia_uvm module; mismatch of CUDA/driver > versions, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6206) RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object
[
https://issues.apache.org/jira/browse/YARN-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881874#comment-15881874
]
Hadoop QA commented on YARN-6206:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 19s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
34s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.timeline.webapp.TestTimelineWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6206 |
| JIRA Patch URL |
https://issues.a
[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881866#comment-15881866
]
Hadoop QA commented on YARN-1728:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-1728 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854366/YARN-1728-branch-2.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aac500b023e2 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 40bc9e7 |
| Default Java | 1.7.0_121 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_121
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 |
| findbugs | v3.0.0 |
| JDK v1.7.0_121 Test Results
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881854#comment-15881854
]
Sunil G commented on YARN-6207:
---
Thanks [~bibinchundatt] for the clarification. I suppose you were mentioning
something like below. Seems fine for me.
{code}
if (!app.isStopped()) {
source.finishApplicationAttempt(app, sourceQueueName);
// Submit to a new queue
dest.submitApplicationAttempt(app, user);
}
// Finish app & update metrics
app.move(dest);
}
source.appFinished();
source.getParent().finishApplication(appId, user);
application.setQueue(dest);
{code}
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881843#comment-15881843 ] Sean Po edited comment on YARN-5385 at 2/24/17 3:03 AM: The two test failures are known test failures tracked in JIRA: YARN-5548 - org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart testFinishedAppRemovalAfterRMRestart YARN-5816 - org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer testCancelWithMultipleAppSubmissions was (Author: seanpo03): The two test failures are: > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6230) Failing unit test TestFairScheduler.testMoveWouldViolateMaxResourcesConstraints
Miklos Szegedi created YARN-6230:
Summary: Failing unit test
TestFairScheduler.testMoveWouldViolateMaxResourcesConstraints
Key: YARN-6230
URL: https://issues.apache.org/jira/browse/YARN-6230
Project: Hadoop YARN
Issue Type: Bug
Reporter: Miklos Szegedi
Priority: Minor
I have run into this in one of the job runs:
{code}
Tests run: 92, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.683 sec <<<
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
testMoveWouldViolateMaxResourcesConstraints(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
Time elapsed: 0.115 sec <<< ERROR!
java.lang.Exception: Unexpected exception,
expected but
was
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testMoveWouldViolateMaxResourcesConstraints(TestFairScheduler.java:4533)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881843#comment-15881843 ] Sean Po commented on YARN-5385: --- The two test failures are: > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6172) TestFSAppStarvation fails on trunk
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881841#comment-15881841
]
Miklos Szegedi commented on YARN-6172:
--
Both failing unit tests are flaky.
> TestFSAppStarvation fails on trunk
> --
>
> Key: YARN-6172
> URL: https://issues.apache.org/jira/browse/YARN-6172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Varun Saxena
>Assignee: Miklos Szegedi
> Attachments: YARN-6172.000.patch
>
>
> Refer to test report
> https://builds.apache.org/job/PreCommit-YARN-Build/14882/testReport/
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.verifyLeafQueueStarvation(TestFSAppStarvation.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:106)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[ https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881837#comment-15881837 ] Xuan Gong commented on YARN-6228: - +1 LGTM. Will commit it shortly. > EntityGroupFSTimelineStore should allow configurable cache stores. > --- > > Key: YARN-6228 > URL: https://issues.apache.org/jira/browse/YARN-6228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-6228-trunk.001.patch, YARN-6228-trunk.002.patch > > > We should allow users to config which cache store to use for > EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated YARN-1728:
-
Attachment: YARN-1728-branch-2.003.patch
[~haibochen] Thanks a lot for your comments.
Upload v3 patch.
> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch,
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch
>
>
> For example, going to the job history server page
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId:
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent
> encoding.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6143) Fix incompatible issue caused by YARN-3583
[ https://issues.apache.org/jira/browse/YARN-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881830#comment-15881830 ] Sunil G commented on YARN-6143: --- Thanks [~wangda] for review and commit. Thanks [~djp].. > Fix incompatible issue caused by YARN-3583 > -- > > Key: YARN-6143 > URL: https://issues.apache.org/jira/browse/YARN-6143 > Project: Hadoop YARN > Issue Type: Sub-task > Components: rolling upgrade >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: YARN-6143.0001.patch, YARN-6143.0002.patch, > YARN-6143.0003.patch, YARN-6143.0004.patch, YARN-6143.0005.patch, > YARN-6143.branch-2.0005.patch, YARN-6143.branch-2.8.0005.patch, > YARN-6143.branch-2.8.0006.patch > > > As mentioned by comment: > https://issues.apache.org/jira/browse/YARN-6142?focusedCommentId=15852009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15852009. > We need to fix the incompatible issue caused by YARN-3583. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6229) resource manager web UI display BUG
gehaijiang created YARN-6229: Summary: resource manager web UI display BUG Key: YARN-6229 URL: https://issues.apache.org/jira/browse/YARN-6229 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.1 Environment: hadoop 2.7.1 Reporter: gehaijiang resourcemanager web UI display bug： Memory Used -3.44TB Containers Running -2607 VCores Used -2607 Lost Nodes 173 These numbers are not correct。 Cluster Metrics Apps Submitted Apps PendingApps RunningApps Completed Containers Running Memory Used Memory TotalMemory Reserved VCores Used VCores TotalVCores Reserved Active NodesDecommissioned NodesLost Nodes Unhealthy Nodes Rebooted Nodes 3027432 0 20 3027412 -2607 -3.44 TB9.70 TB 0 B -2607 72400 181 0 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6172) TestFSAppStarvation fails on trunk
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881814#comment-15881814
]
Hadoop QA commented on YARN-6172:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 51s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6172 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854347/YARN-6172.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4e1a62fa04f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 132f758 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15070/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15070/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15070/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestFSAppStarvation fails on trunk
> --
>
> Key: YARN-6172
> URL: https://i
[jira] [Commented] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[
https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881809#comment-15881809
]
Hadoop QA commented on YARN-6228:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
48s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6228 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854335/YARN-6228-trunk.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 2d12c4912088 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 132f758 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15068/testReport/ |
| modules | C: ha
[jira] [Updated] (YARN-6218) TestAMRMClient fails with fair scheduler
[ https://issues.apache.org/jira/browse/YARN-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6218: - Attachment: YARN-6218.001.patch Thank you, [~haibochen] for the review. I updated the patch. > TestAMRMClient fails with fair scheduler > > > Key: YARN-6218 > URL: https://issues.apache.org/jira/browse/YARN-6218 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6218.000.patch, YARN-6218.001.patch > > > We ran into this issue on v2. Allocation does not happen in the specified > amount of time. > Error Message > expected:<2> but was:<0> > Stacktrace > java.lang.AssertionError: expected:<2> but was:<0> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientMatchStorage(TestAMRMClient.java:495) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6172) TestFSAppStarvation fails on trunk
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi reassigned YARN-6172:
Assignee: Miklos Szegedi
> TestFSAppStarvation fails on trunk
> --
>
> Key: YARN-6172
> URL: https://issues.apache.org/jira/browse/YARN-6172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Varun Saxena
>Assignee: Miklos Szegedi
> Attachments: YARN-6172.000.patch
>
>
> Refer to test report
> https://builds.apache.org/job/PreCommit-YARN-Build/14882/testReport/
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.verifyLeafQueueStarvation(TestFSAppStarvation.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:106)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881721#comment-15881721
]
Tao Jie commented on YARN-6215:
---
[~kasha] thank you for you comment!
I understand your concern. It may bring some performance loss if we add a
read-lock, but it is risky if we don't. Doing preemption when fairshare of
queues is incomplete would not only miss containers that should be preempted,
but also may preempt containers that should not be preempted by mistake. It
would be unpredictable.
In earlier FS code, updating fairshare and preempting containers took place in
one thread, so I think a readlock here would not make the performance worse.
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
> Attachments: YARN-6215.001.patch
>
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959-branch-2.002.patch Fixing javadoc warning, some checkstyles and the SLS testcase. The remaining test failures are not related and the TestCapacityScheduler works fine for me. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5959.005.patch, YARN-5959-branch-2.001.patch, > YARN-5959-branch-2.002.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch, > YARN-5959-YARN-5085.001.patch, YARN-5959-YARN-5085.002.patch, > YARN-5959-YARN-5085.003.patch, YARN-5959-YARN-5085.004.patch, > YARN-5959-YARN-5085.005.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar to YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6172) TestFSAppStarvation fails on trunk
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-6172:
-
Attachment: YARN-6172.000.patch
> TestFSAppStarvation fails on trunk
> --
>
> Key: YARN-6172
> URL: https://issues.apache.org/jira/browse/YARN-6172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Varun Saxena
> Attachments: YARN-6172.000.patch
>
>
> Refer to test report
> https://builds.apache.org/job/PreCommit-YARN-Build/14882/testReport/
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.verifyLeafQueueStarvation(TestFSAppStarvation.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:106)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6172) TestFSAppStarvation fails on trunk
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881712#comment-15881712
]
Miklos Szegedi edited comment on YARN-6172 at 2/24/17 1:23 AM:
---
Thank you, [~varun_saxena] for reporting this.
I was able to repro the scenario above. There are two issues here.
First, the update thread resets the queue demand and adds each application
demand to it one by one every time it runs without locking. Whenever this value
is sampled, the test compares it with the expected value. However, if we have
not finished with the update, this can be 0 or anything less than the actual
demand.
A different unrelated issue is that the test actually calls {{Thread.yield()}}
instead of properly waiting for the expected application count value to
propagate. I will send out a patch soon.
{code}
@Override
public void updateDemand() {
// Compute demand by iterating through apps in the queue
// Limit demand to maxResources
demand = Resources.createResource(0);
readLock.lock();
try {
for (FSAppAttempt sched : runnableApps) {
updateDemandForApp(sched);
}
for (FSAppAttempt sched : nonRunnableApps) {
updateDemandForApp(sched);
}
} finally {
readLock.unlock();
}
// Cap demand to maxShare to limit allocation to maxShare
demand = Resources.componentwiseMin(demand, maxShare);
if (LOG.isDebugEnabled()) {
LOG.debug("The updated demand for " + getName() + " is " + demand
+ "; the max is " + maxShare);
LOG.debug("The updated fairshare for " + getName() + " is "
+ getFairShare());
}
}
private void updateDemandForApp(FSAppAttempt sched) {
sched.updateDemand();
Resource toAdd = sched.getDemand();
if (LOG.isDebugEnabled()) {
LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
+ "; Total resource demand for " + getName() + " now "
+ demand);
}
demand = Resources.add(demand, toAdd);
}
{code}
was (Author: miklos.szeg...@cloudera.com):
I was able to repro the scenario above. There are two issues here.
First, the update thread resets the queue demand and adds each application
demand to it one by one every time it runs without locking. Whenever this value
is sampled, the test compares it with the expected value. However, if we have
not finished with the update, this can be 0 or anything less than the actual
demand.
A different unrelated issue is that the test actually calls {{Thread.yield()}}
instead of properly waiting for the expected application count value to
propagate. I will send out a patch soon.
{code}
@Override
public void updateDemand() {
// Compute demand by iterating through apps in the queue
// Limit demand to maxResources
demand = Resources.createResource(0);
readLock.lock();
try {
for (FSAppAttempt sched : runnableApps) {
updateDemandForApp(sched);
}
for (FSAppAttempt sched : nonRunnableApps) {
updateDemandForApp(sched);
}
} finally {
readLock.unlock();
}
// Cap demand to maxShare to limit allocation to maxShare
demand = Resources.componentwiseMin(demand, maxShare);
if (LOG.isDebugEnabled()) {
LOG.debug("The updated demand for " + getName() + " is " + demand
+ "; the max is " + maxShare);
LOG.debug("The updated fairshare for " + getName() + " is "
+ getFairShare());
}
}
private void updateDemandForApp(FSAppAttempt sched) {
sched.updateDemand();
Resource toAdd = sched.getDemand();
if (LOG.isDebugEnabled()) {
LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
+ "; Total resource demand for " + getName() + " now "
+ demand);
}
demand = Resources.add(demand, toAdd);
}
{code}
> TestFSAppStarvation fails on trunk
> --
>
> Key: YARN-6172
> URL: https://issues.apache.org/jira/browse/YARN-6172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Varun Saxena
> Attachments: YARN-6172.000.patch
>
>
> Refer to test report
> https://builds.apache.org/job/PreCommit-YARN-Build/14882/testReport/
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.verifyLeafQueueStarvation(TestFSAppStarvation.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:106)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---
[jira] [Commented] (YARN-6172) TestFSAppStarvation fails on trunk
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881712#comment-15881712
]
Miklos Szegedi commented on YARN-6172:
--
I was able to repro the scenario above. There are two issues here.
First, the update thread resets the queue demand and adds each application
demand to it one by one every time it runs without locking. Whenever this value
is sampled, the test compares it with the expected value. However, if we have
not finished with the update, this can be 0 or anything less than the actual
demand.
A different unrelated issue is that the test actually calls {{Thread.yield()}}
instead of properly waiting for the expected application count value to
propagate. I will send out a patch soon.
{code}
@Override
public void updateDemand() {
// Compute demand by iterating through apps in the queue
// Limit demand to maxResources
demand = Resources.createResource(0);
readLock.lock();
try {
for (FSAppAttempt sched : runnableApps) {
updateDemandForApp(sched);
}
for (FSAppAttempt sched : nonRunnableApps) {
updateDemandForApp(sched);
}
} finally {
readLock.unlock();
}
// Cap demand to maxShare to limit allocation to maxShare
demand = Resources.componentwiseMin(demand, maxShare);
if (LOG.isDebugEnabled()) {
LOG.debug("The updated demand for " + getName() + " is " + demand
+ "; the max is " + maxShare);
LOG.debug("The updated fairshare for " + getName() + " is "
+ getFairShare());
}
}
private void updateDemandForApp(FSAppAttempt sched) {
sched.updateDemand();
Resource toAdd = sched.getDemand();
if (LOG.isDebugEnabled()) {
LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
+ "; Total resource demand for " + getName() + " now "
+ demand);
}
demand = Resources.add(demand, toAdd);
}
{code}
> TestFSAppStarvation fails on trunk
> --
>
> Key: YARN-6172
> URL: https://issues.apache.org/jira/browse/YARN-6172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Varun Saxena
>
> Refer to test report
> https://builds.apache.org/job/PreCommit-YARN-Build/14882/testReport/
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.verifyLeafQueueStarvation(TestFSAppStarvation.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:106)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6206) RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object
[
https://issues.apache.org/jira/browse/YARN-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-6206:
--
Attachment: YARN-6206.3.patch
> RequestHedgingRMFailoverProvider may have concurrency issues when accessing
> the same request object
> ---
>
> Key: YARN-6206
> URL: https://issues.apache.org/jira/browse/YARN-6206
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-6206.1.patch, YARN-6206.2.patch, YARN-6206.3.patch
>
>
> Here, since multiple threads are accessing the same request object - 'args',
> and the request protobuf implementation is not synchronized. That caused
> errors like below:
> {code}
>Callable c = new Callable() {
> @Override
> public Object call() throws Exception {
> return method.invoke(pInfo.proxy, args);
> }
> };
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.ArrayList.elementData(ArrayList.java:418)
> at java.util.ArrayList.get(ArrayList.java:431)
> at
> com.google.protobuf.LazyStringArrayList.getByteString(LazyStringArrayList.java:157)
> at
> com.google.protobuf.UnmodifiableLazyStringList.getByteString(UnmodifiableLazyStringList.java:66)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationsRequestProto.getSerializedSize(YarnServiceProtos.java:17070)
> at
> com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.write(ProtobufRpcEngine.java:356)
> at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1059)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1398)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy17.getApplications(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:251)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:282)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplications(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[ https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-6228: Attachment: YARN-6228-trunk.002.patch I cannot reproduce the failures locally. Try again... > EntityGroupFSTimelineStore should allow configurable cache stores. > --- > > Key: YARN-6228 > URL: https://issues.apache.org/jira/browse/YARN-6228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-6228-trunk.001.patch, YARN-6228-trunk.002.patch > > > We should allow users to config which cache store to use for > EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881649#comment-15881649
]
Hadoop QA commented on YARN-5892:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 31s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5892 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854316/YARN-5892.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fce68500d71f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 694e680 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15066/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15066/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15066/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> UR
[jira] [Commented] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[
https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881643#comment-15881643
]
Hadoop QA commented on YARN-6228:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
27s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-server-timeline-pluginstorage in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-server-timeline-pluginstorage in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-server-timeline-pluginstorage in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s{color}
| {color:red} hadoop-yarn-server-timeline-pluginstorage in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6228 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854317/YARN-6228-trunk.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux bc3
[jira] [Commented] (YARN-6042) Dump scheduler and queue state information into FairScheduler DEBUG log
[
https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881636#comment-15881636
]
Ray Chiang commented on YARN-6042:
--
Very minor nit:
The result of this part of code:
{quote}
rootMetrics.getAvailableMB(), rootMetrics.getAvailableVirtualCores()) +
rootQueue.dumpState());
{quote}
There is no separation between the scheduler and the queue states. From my
sample output, the part in red looks a little odd:
{quote}
2017-02-23 14:53:29,644 DEBUG fair.FairScheduler: FairScheduler state: Cluster
Capacity: Allocations:
Availability: {{color}Name: root, Weight:
, Policy: fair, FairShare: , SteadyFairShare: ,
{quote}
I'd suggest adding two spaces and possibly a label like the rest of the
scheduler state?
> Dump scheduler and queue state information into FairScheduler DEBUG log
> ---
>
> Key: YARN-6042
> URL: https://issues.apache.org/jira/browse/YARN-6042
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6042.001.patch, YARN-6042.002.patch,
> YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch,
> YARN-6042.006.patch, YARN-6042.007.patch
>
>
> To improve the debugging of scheduler issues it would be a big improvement to
> be able to dump the scheduler state into a log on request.
> The Dump the scheduler state at a point in time would allow debugging of a
> scheduler that is not hung (deadlocked) but also not assigning containers.
> Currently we do not have a proper overview of what state the scheduler and
> the queues are in and we have to make assumptions or guess
> The scheduler and queue state needed would include (not exhaustive):
> - instantaneous and steady fair share (app / queue)
> - AM share and resources
> - weight
> - app demand
> - application run state (runnable/non runnable)
> - last time at fair/min share
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6143) Fix incompatible issue caused by YARN-3583
[ https://issues.apache.org/jira/browse/YARN-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881618#comment-15881618 ] Junping Du commented on YARN-6143: -- Thanks [~leftnoteasy] and [~sunilg] for working on it. Hope we don't have more 2.8 blockers now. :) > Fix incompatible issue caused by YARN-3583 > -- > > Key: YARN-6143 > URL: https://issues.apache.org/jira/browse/YARN-6143 > Project: Hadoop YARN > Issue Type: Sub-task > Components: rolling upgrade >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: YARN-6143.0001.patch, YARN-6143.0002.patch, > YARN-6143.0003.patch, YARN-6143.0004.patch, YARN-6143.0005.patch, > YARN-6143.branch-2.0005.patch, YARN-6143.branch-2.8.0005.patch, > YARN-6143.branch-2.8.0006.patch > > > As mentioned by comment: > https://issues.apache.org/jira/browse/YARN-6142?focusedCommentId=15852009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15852009. > We need to fix the incompatible issue caused by YARN-3583. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881614#comment-15881614
]
Wangda Tan commented on YARN-5892:
--
Hi [~eepayne],
Apologize for my late responses, finally get some time to look at this.
I'm a little concerned about semantics of this feature. (I've to admit that
existing user-limit semantics is not perfect first) For existing MULP, to me it
has two guarantees:
#1, if there're N (N <= 100 / MULP) users are consuming resource in a queue,
each of them can get at least MULP / 100 * queue-configured-capacity.
#2, when doing preemption, if an user uses <= MULP / 100 *
queue-configured-capacity, we will not preempt resource from such users.
If we introduced this feature, per my understanding, #1 will be break. For
example, a queue with MULP = 33, and user=admin has MULP = 75. Assume we have 3
users: Jack, Alice, Admin are running jobs in the queue, there's no guarantee
that any user can use at much as their MULP.
To make a more clear semantics, in my mind there're some alternative solutions:
a. Create queue just for such vip users, and we can plan resources, (for
example root.users.admin = 30%, and root.users.others = 50%).
b. In addition to that, special queue mapping rules can be implemented based on
YARN-3635, which can do more flexible queue mappings for use cases like vip
users.
Please let me know you thoughts.
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-5892.001.patch, YARN-5892.002.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6218) TestAMRMClient fails with fair scheduler
[ https://issues.apache.org/jira/browse/YARN-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881606#comment-15881606 ] Haibo Chen commented on YARN-6218: -- Thanks for the patch [~miklos.szeg...@cloudera.com]! Took a cursory look at the patch. 1) In places where sleep() is replaced with waitForNMHeartbeat(), we may want to update the comments as well since " sleep to let NM's heartbeat to RM and trigger allocations" no longer makes sense. 2) At line 1015, the code is check container status from NMs, so it does not need to wait for NM heartbeats to RM. 3) bq. // Wait for fair scheduler update thread We are synchronously calling update() in the test thread, not waiting for the update thread. so maybe we can say triggering fair scheduler update? > TestAMRMClient fails with fair scheduler > > > Key: YARN-6218 > URL: https://issues.apache.org/jira/browse/YARN-6218 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6218.000.patch > > > We ran into this issue on v2. Allocation does not happen in the specified > amount of time. > Error Message > expected:<2> but was:<0> > Stacktrace > java.lang.AssertionError: expected:<2> but was:<0> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientMatchStorage(TestAMRMClient.java:495) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6222) TestFairScheduler.testReservationMetrics is flaky
[ https://issues.apache.org/jira/browse/YARN-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881594#comment-15881594 ] Hudson commented on YARN-6222: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11296 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11296/]) YARN-6222. TestFairScheduler.testReservationMetrics is flaky. (Yufei Gu (kasha: rev 694e680d20dc07f634b539537021b09d9316601c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > TestFairScheduler.testReservationMetrics is flaky > - > > Key: YARN-6222 > URL: https://issues.apache.org/jira/browse/YARN-6222 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6222.001.patch, YARN-6222.002.patch > > > TestFairScheduler.testReservationMetrics is flaky because of timing issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881587#comment-15881587
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| |
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PriorityReservationAgent.createReservation(ReservationId,
String, Plan, ReservationDefinition) does not release lock on all paths At
PriorityReservationAgent.java:not release lock on all paths At
PriorityReservationAgent.java:[line 80] |
| |
org.apache.hadoop.yarn.server.resourcemanager.reserv
[jira] [Commented] (YARN-6222) TestFairScheduler.testReservationMetrics is flaky
[ https://issues.apache.org/jira/browse/YARN-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881558#comment-15881558 ] Yufei Gu commented on YARN-6222: [~kasha], thanks for the review and commit. > TestFairScheduler.testReservationMetrics is flaky > - > > Key: YARN-6222 > URL: https://issues.apache.org/jira/browse/YARN-6222 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6222.001.patch, YARN-6222.002.patch > > > TestFairScheduler.testReservationMetrics is flaky because of timing issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[ https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-6228: Attachment: YARN-6228-trunk.001.patch Patch to make cache stores configurable. > EntityGroupFSTimelineStore should allow configurable cache stores. > --- > > Key: YARN-6228 > URL: https://issues.apache.org/jira/browse/YARN-6228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-6228-trunk.001.patch > > > We should allow users to config which cache store to use for > EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-5892:
-
Attachment: YARN-5892.002.patch
Uploading new patch to address the javadoc and findbug warnings. I also
modified this patch to be refreshable.
The unit test ({{TestRMRestart}}) is failing intermittently on trunk with and
without this patch.
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-5892.001.patch, YARN-5892.002.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881523#comment-15881523
]
Sangjin Lee commented on YARN-6027:
---
In addition to [~varun_saxena]'s comments, I have a couple of comments.
(FlowActivityRowKey.java)
- In addition to Varun's comment above, in fact the new {{decode()}} method
seems largely redundant with {{FlowActivityRowKeyConverter.decode()}}. Is there
a reason not to simply use it? The same comment goes for
{{getRowKeyAsString()}}. Converters are how we apply encoding and decoding
consistently, and we should try to stick to that as much as possible.
(AbstractHBaseTestBase.java)
- I am a little curious about this refactoring. Is the idea to split the
current reader unit tests based on this abstract class? At least for now, only
{{TestTimelineReaderWebServicesHBaseStorage}} is using it.
- Is the name the best? Also, if this is specifically for reader unit tests, it
might be better to place it in the reader package and indicate that in the
class name also; e.g. {{AbstractTimelineReaderHBaseTestBase}}
> Support fromid(offset) filter for /flows API
>
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch,
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch,
> YARN-6027-YARN-5355.0004.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar
> filter for flows/flowRun apps and flow run and flow as well.
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6222) TestFairScheduler.testReservationMetrics is flaky
[ https://issues.apache.org/jira/browse/YARN-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881520#comment-15881520 ] Yufei Gu commented on YARN-6222: [~kasha], after patch, I ran it 200 times and it didn't fail. > TestFairScheduler.testReservationMetrics is flaky > - > > Key: YARN-6222 > URL: https://issues.apache.org/jira/browse/YARN-6222 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6222.001.patch, YARN-6222.002.patch > > > TestFairScheduler.testReservationMetrics is flaky because of timing issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
Li Lu created YARN-6228: --- Summary: EntityGroupFSTimelineStore should allow configurable cache stores. Key: YARN-6228 URL: https://issues.apache.org/jira/browse/YARN-6228 Project: Hadoop YARN Issue Type: Sub-task Reporter: Li Lu Assignee: Li Lu We should allow users to config which cache store to use for EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) FairScheduler: Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881449#comment-15881449
]
Yufei Gu commented on YARN-6175:
Thanks [~kasha] for the review and commit.
> FairScheduler: Negative vcore for resource needed to preempt
>
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.8.0
>
> Attachments: YARN-6175.001.patch, YARN-6175.branch-2.8.002.patch,
> YARN-6175.branch-2.8.003.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6210) FS: Node reservations can interfere with preemption
[ https://issues.apache.org/jira/browse/YARN-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-6210: --- Fix Version/s: 3.0.0-alpha3 > FS: Node reservations can interfere with preemption > --- > > Key: YARN-6210 > URL: https://issues.apache.org/jira/browse/YARN-6210 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6210.1.patch, YARN-6210.2.patch, YARN-6210.3.patch, > YARN-6210.4.patch > > > Today, on a saturated cluster, apps with pending demand reserve nodes. A new > app might not be able to preempt resources because these nodes are already > reserved. This can be reproduced by the example in YARN-6151. > Since node reservations are to prevent starvation of apps requesting large > containers, triggering these reservations only on starved applications would > avoid this situation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6194) Cluster capacity in SchedulingPolicy is updated only on allocation file reload
[
https://issues.apache.org/jira/browse/YARN-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6194:
---
Fix Version/s: 3.0.0-alpha3
> Cluster capacity in SchedulingPolicy is updated only on allocation file reload
> --
>
> Key: YARN-6194
> URL: https://issues.apache.org/jira/browse/YARN-6194
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6194.001.patch
>
>
> Some of the {{SchedulingPolicy}} methods need cluster capacity which is set
> using {{#initialize}} today. However, {{initialize()}} is called only on
> allocation reload. If nodes are added between reloads, the cluster capacity
> is not considered until the next reload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6175) FairScheduler: Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6175:
---
Summary: FairScheduler: Negative vcore for resource needed to preempt
(was: Negative vcore for resource needed to preempt)
> FairScheduler: Negative vcore for resource needed to preempt
>
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch, YARN-6175.branch-2.8.002.patch,
> YARN-6175.branch-2.8.003.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881397#comment-15881397
]
Haibo Chen commented on YARN-1728:
--
Thanks [~yuanbo] for your update! A few more nits.
1) bq. // Guice-3.0 doesn't support encoded path info,
Support seems a little vague. We can be more concrete by saying like Guice-3.0
does not decode paths that are encoded
2) Can we rename testEncodedText to testEncodedUrl?
3) app.stop() is called in the final block in all other test methods, we should
follow that practice as well.
> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch,
> YARN-1728-branch-2.002.patch
>
>
> For example, going to the job history server page
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId:
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent
> encoding.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v005.patch Thanks [~curino] for the review! I have addressed most of the comments you have given me. My comments below: 1. The CapacitySchedulerConfiguration implements the get by queue method. 2. I have changed the words 'accommodate for' in code and comments to 'make room for'. 3. In our clusters, we currently do not see many priority inversions. For our use, it seems that this log won't be too verbose, and having the logs will be more beneficial than harmful. 4. The makeRoomForReservation method is abstract for PriorityReservationAgent because subclasses will want to determine 2 things. 1, the order for deletion, and 2, the order for re-addition. I have documented this in the java doc to make it more clear that makeRoomForReservation is meant to return an ordered list of reservation allocations. 5. This problem will require a lot of work, and it may be more appropriate for this to be completed in a different task. I have created YARN-6227 to track this feature. 7. I have added locks around the implementation of the ReservationAgent API for the PriorityReservationAgent. I understand this solution may not be the best in the event that we get a lot of multi-threaded access in a short period of time, but a more appropriate solution will be complicated, and it may be better if it is done in a separate task. YARN-6226 was created to track this improvement. 10. In RMWebService, if the priority is not given, it will be set to an undefined value. PriorityReservationAgent converts undefined priority to the lowest possible priority. 11. I do have tests that check invariants. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6190) Bug in LocalityMulticastAMRMProxyPolicy argument validation
[
https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881278#comment-15881278
]
Botong Huang edited comment on YARN-6190 at 2/23/17 10:09 PM:
--
Thanks [~curino] for the feed back. For 1), the Federation Interceptor code
doesn't have synchronization on allocate calls. It can be processing allocate
calls concurrently. So I believe we do need to cover concurrency here.
I just uploaded the v2 patch. Instead of using synchronization on the entire
object, we can create a new {{AllocationBookkeeper}} object per
splitResourceRequests call.
I also modified the way we compute the weights for ANY ( * ) request with a
request id. Specifically, rather than divide the local weight by the total
weight of all request ids, we should divide by the total weight associated with
the specific request id.
Please take a look. Thanks!
was (Author: botong):
Thanks [~curino] for the feed back. For 1), the Federation Interceptor code
doesn't have synchronization on allocate calls. It can be processing allocate
calls concurrently. So I believe we do need to cover concurrency here.
I just uploaded the v2 patch. Instead of using synchronization on the entire
object, we can create a new {{AllocationBookkeeper}} object per
splitResourceRequests call.
I also modified the way we compute the weights for ANY (*) request with a
request id. Specifically, rather than divide the local weight by the total
weight of all request ids, we should divide by the total weight associated with
the specific request id.
Please take a look. Thanks!
> Bug in LocalityMulticastAMRMProxyPolicy argument validation
> ---
>
> Key: YARN-6190
> URL: https://issues.apache.org/jira/browse/YARN-6190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6190-YARN-2915.v1.patch,
> YARN-6190-YARN-2915.v2.patch
>
>
> A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition
> check, along with misc cleanups.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6190) Bug in LocalityMulticastAMRMProxyPolicy argument validation
[
https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881278#comment-15881278
]
Botong Huang edited comment on YARN-6190 at 2/23/17 10:09 PM:
--
Thanks [~curino] for the feed back. For 1), the Federation Interceptor code
doesn't have synchronization on allocate calls. It can be processing allocate
calls concurrently. So I believe we do need to cover concurrency here.
I just uploaded the v2 patch. Instead of using synchronization on the entire
object, we can create a new {{AllocationBookkeeper}} object per
splitResourceRequests call.
I also modified the way we compute the weights for ANY (*) request with a
request id. Specifically, rather than divide the local weight by the total
weight of all request ids, we should divide by the total weight associated with
the specific request id.
Please take a look. Thanks!
was (Author: botong):
Thanks [~curino] for the feed back. For 1), the Federation Interceptor code
doesn't have synchronization on allocate calls. It can be processing allocate
calls concurrently. So I believe we do need to cover concurrency here.
I just uploaded the v2 patch. Instead of using synchronization on the entire
object, we can create a new {{AllocationBookkeeper}} object per
splitResourceRequests call.
I also modified the way we compute the weights for any request with a request
id. Specifically, rather than divide the local weight by the total weight for
all request ids, we should divide by the total weight associated with the
specific request id.
Please take a look. Thanks!
> Bug in LocalityMulticastAMRMProxyPolicy argument validation
> ---
>
> Key: YARN-6190
> URL: https://issues.apache.org/jira/browse/YARN-6190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6190-YARN-2915.v1.patch,
> YARN-6190-YARN-2915.v2.patch
>
>
> A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition
> check, along with misc cleanups.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6227) Information about reservations that are yielded by higher priority reservations need to be surfaced to the client
Sean Po created YARN-6227: - Summary: Information about reservations that are yielded by higher priority reservations need to be surfaced to the client Key: YARN-6227 URL: https://issues.apache.org/jira/browse/YARN-6227 Project: Hadoop YARN Issue Type: Task Reporter: Sean Po YARN-5385 introduces a PriorityReservationAgent that can be used to yield lower priority reservations so that higher priority reservations may fit if it initially could not be created in the plan. In the event that a low priority reservation is yielded, the client does not have a way to know prior to using the reservation that it has been removed. This JIRA aims to surface such information to the client, so that it may be able to act appropriately to its lost reservation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881335#comment-15881335
]
Hadoop QA commented on YARN-5959:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
37s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 54s{color} | {color:orange} root: The patch generated 38 new + 1600
unchanged - 18 fixed = 1638 total (was 1618) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121
with JDK v1.8.0_121 generated 1 new + 907 unchanged - 1 fixed = 908 total (was
908) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-server-common in the patch passed with
JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 5s{color}
| {color:red} hadoop-yarn-server-resourceman
[jira] [Created] (YARN-6226) Improve concurrency model in PriorityReservationAgent
Sean Po created YARN-6226: - Summary: Improve concurrency model in PriorityReservationAgent Key: YARN-6226 URL: https://issues.apache.org/jira/browse/YARN-6226 Project: Hadoop YARN Issue Type: Task Components: resourcemanager Reporter: Sean Po Assignee: Sean Po YARN-5385 introduces a PriorityReservationAgent that can be used to yield lower priority reservations so that higher priority reservations may fit if it initially could not be created in the plan. In YARN-5385, a write lock was used in the createReservation, updateReservation and deleteReservation methods to ensure that multi-threaded access will not be a problem for the new agent. This solution may be problematic if many multi-threaded reservation operations are made in a short period of time due to the heaviness of the createReservation and updateReservation methods. This JIRA aims to improve the concurrency model of the PriorityReservationAgent by replacing the write lock with a more suitable solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881326#comment-15881326
]
Hadoop QA commented on YARN-5959:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
0s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
29s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-mapreduce-client-app in branch-2 failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 55s{color} | {color:orange} root: The patch generated 38 new + 1600
unchanged - 18 fixed = 1638 total (was 1618) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
27s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121
with JDK v1.8.0_121 generated 1 new + 907 unchanged - 1 fixed = 908 total (was
908) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-server-common in the patch passed with
JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 16s{color}
| {color:red} hadoo
[jira] [Commented] (YARN-6190) Bug in LocalityMulticastAMRMProxyPolicy argument validation
[
https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881278#comment-15881278
]
Botong Huang commented on YARN-6190:
Thanks [~curino] for the feed back. For 1), the Federation Interceptor code
doesn't have synchronization on allocate calls. It can be processing allocate
calls concurrently. So I believe we do need to cover concurrency here.
I just uploaded the v2 patch. Instead of using synchronization on the entire
object, we can create a new {{AllocationBookkeeper}} object per
splitResourceRequests call.
I also modified the way we compute the weights for any request with a request
id. Specifically, rather than divide the local weight by the total weight for
all request ids, we should divide by the total weight associated with the
specific request id.
Please take a look. Thanks!
> Bug in LocalityMulticastAMRMProxyPolicy argument validation
> ---
>
> Key: YARN-6190
> URL: https://issues.apache.org/jira/browse/YARN-6190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6190-YARN-2915.v1.patch,
> YARN-6190-YARN-2915.v2.patch
>
>
> A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition
> check, along with misc cleanups.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6190) Bug in LocalityMulticastAMRMProxyPolicy argument validation
[ https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6190: --- Attachment: YARN-6190-YARN-2915.v2.patch > Bug in LocalityMulticastAMRMProxyPolicy argument validation > --- > > Key: YARN-6190 > URL: https://issues.apache.org/jira/browse/YARN-6190 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6190-YARN-2915.v1.patch, > YARN-6190-YARN-2915.v2.patch > > > A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition > check, along with misc cleanups. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6109) Add an ability to convert ChildQueue to ParentQueue
[ https://issues.apache.org/jira/browse/YARN-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881113#comment-15881113 ] Wangda Tan commented on YARN-6109: -- [~subru], make sense, are you OK with moving the two fixes you mentioned to a separate JIRA? > Add an ability to convert ChildQueue to ParentQueue > --- > > Key: YARN-6109 > URL: https://issues.apache.org/jira/browse/YARN-6109 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6109.1.patch, YARN-6109.2.patch, > YARN-6109.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4954) TestYarnClient.testReservationAPIs fails on machines with less than 4 GB available memory
[
https://issues.apache.org/jira/browse/YARN-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881097#comment-15881097
]
Wangda Tan commented on YARN-4954:
--
[~ebadger],
I believe YARN-5389 is the issue. Since we're using MiniYarnCluster which by
default inits host's memory to 4G, I don't think MiniYarnCluster tests related
to host's actual physical memory. Let's wait for [~GergelyNovak]'s comment
before closing this ticket.
> TestYarnClient.testReservationAPIs fails on machines with less than 4 GB
> available memory
> -
>
> Key: YARN-4954
> URL: https://issues.apache.org/jira/browse/YARN-4954
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Critical
> Attachments: YARN-4954.001.patch, YARN-4954.002.patch
>
>
> TestYarnClient.testReservationAPIs sometimes fails with this error:
> {noformat}
> java.lang.AssertionError:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1254)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:457)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:515)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2422)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2418)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2416)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.IterativePlanner.computeJobAllocation(IterativePlanner.java:151)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:64)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1237)
> ... 10 more
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationAPIs(TestYarnClient.java:1227)
> {noformat}
> This is caused by really not having enough available memory to complete the
> reservation (4 * 1024 MB). In my opinion lowering the required memory (either
> by lowering the number of containers to 2, or the memory to 512 MB) would
> make the test more stable.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5410) Bootstrap Router module
[ https://issues.apache.org/jira/browse/YARN-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881083#comment-15881083 ] Wangda Tan commented on YARN-5410: -- One question to [~giovanni.fumarola]/[~subru]: I think I understand the purpose of Router. However, since it implements ClientRMService, it behaves as gateway of RM. I'm not sure if any logics could be shared with resource-manager module. If you're confident with router will be independent from RM, I'm fine with creating this sub module. > Bootstrap Router module > --- > > Key: YARN-5410 > URL: https://issues.apache.org/jira/browse/YARN-5410 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-5410-YARN-2915-v1.patch, > YARN-5410-YARN-2915-v2.patch, YARN-5410-YARN-2915-v3.patch, > YARN-5410-YARN-2915-v4.patch, YARN-5410-YARN-2915-v5.patch, > YARN-5410-YARN-2915-v6.patch > > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate ResourceManager(s). This > JIRA tracks the creation of a new sub-module for the Router. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4954) TestYarnClient.testReservationAPIs fails on machines with less than 4 GB available memory
[
https://issues.apache.org/jira/browse/YARN-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881074#comment-15881074
]
Eric Badger edited comment on YARN-4954 at 2/23/17 7:29 PM:
[~leftnoteasy], YARN-5389 fixed the timeout failures related to the NMs not
heartbeating quickly enough to update the RM's capacity. But this JIRA is more
about the requests being too high in general. I can't get this test to fail
anymore locally and I have forgotten how exactly it was failing for me in the
past. I might be getting Jiras mixed up in my head, but I remember something
about capacities being updated in different places and not propagating quickly
enough once we got past the test check. However, that doesn't seem to be
mentioned here, so it is quite possible that I'm thinking of a different Jira.
[~GergelyNovak] addressed the issue of machines/VMs with less than 4GB of
memory and confirmed that it still worked, so I'll take his word for it. I'm ok
closing this and reopening if the issue shows again.
Edit: Pretty sure the Jira I'm mixing this one up with about the capacity not
getting propagated is YARN-5994
was (Author: ebadger):
[~leftnoteasy], YARN-5389 fixed the timeout failures related to the NMs not
heartbeating quickly enough to update the RM's capacity. But this JIRA is more
about the requests being too high in general. I can't get this test to fail
anymore locally and I have forgotten how exactly it was failing for me in the
past. I might be getting Jiras mixed up in my head, but I remember something
about capacities being updated in different places and not propagating quickly
enough once we got past the test check. However, that doesn't seem to be
mentioned here, so it is quite possible that I'm thinking of a different Jira.
[~GergelyNovak] addressed the issue of machines/VMs with less than 4GB of
memory and confirmed that it still worked, so I'll take his word for it. I'm ok
closing this and reopening if the issue shows again.
> TestYarnClient.testReservationAPIs fails on machines with less than 4 GB
> available memory
> -
>
> Key: YARN-4954
> URL: https://issues.apache.org/jira/browse/YARN-4954
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Critical
> Attachments: YARN-4954.001.patch, YARN-4954.002.patch
>
>
> TestYarnClient.testReservationAPIs sometimes fails with this error:
> {noformat}
> java.lang.AssertionError:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1254)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:457)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:515)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2422)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2418)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2416)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.IterativePlanner.computeJobAllocation(IterativePlanner.java:151)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:64)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoo
[jira] [Commented] (YARN-4954) TestYarnClient.testReservationAPIs fails on machines with less than 4 GB available memory
[
https://issues.apache.org/jira/browse/YARN-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881074#comment-15881074
]
Eric Badger commented on YARN-4954:
---
[~leftnoteasy], YARN-5389 fixed the timeout failures related to the NMs not
heartbeating quickly enough to update the RM's capacity. But this JIRA is more
about the requests being too high in general. I can't get this test to fail
anymore locally and I have forgotten how exactly it was failing for me in the
past. I might be getting Jiras mixed up in my head, but I remember something
about capacities being updated in different places and not propagating quickly
enough once we got past the test check. However, that doesn't seem to be
mentioned here, so it is quite possible that I'm thinking of a different Jira.
[~GergelyNovak] addressed the issue of machines/VMs with less than 4GB of
memory and confirmed that it still worked, so I'll take his word for it. I'm ok
closing this and reopening if the issue shows again.
> TestYarnClient.testReservationAPIs fails on machines with less than 4 GB
> available memory
> -
>
> Key: YARN-4954
> URL: https://issues.apache.org/jira/browse/YARN-4954
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Critical
> Attachments: YARN-4954.001.patch, YARN-4954.002.patch
>
>
> TestYarnClient.testReservationAPIs sometimes fails with this error:
> {noformat}
> java.lang.AssertionError:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1254)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:457)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:515)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2422)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2418)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2416)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.IterativePlanner.computeJobAllocation(IterativePlanner.java:151)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:64)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1237)
> ... 10 more
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationAPIs(TestYarnClient.java:1227)
> {noformat}
> This is caused by really not having enough available memory to complete the
> reservation (4 * 1024 MB). In my opinion lowering the required memory (either
> by lowering the number of containers to 2, or the memory to 512 MB) would
> make the test more stable.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6216) Unify Container Resizing code paths with Container Updates making it scheduler agnostic
[
https://issues.apache.org/jira/browse/YARN-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881061#comment-15881061
]
Wangda Tan commented on YARN-6216:
--
Hi [~asuresh], the patch looks GREAT and simplified a lot of logics which is
added for container resizing, thanks for working on this!
*Major:*
1) It looks like all container change request needs to create a new
containerId, correct? It's not a critical issue, but could potentially make the
container-id becomes very sparse. I think this should be optimizable.
2) Could you elaborate new changes added to
AbstractYarnScheduler#completedContainer?
3) Can we simplify decrease/demote container process: now it is create a new
RMContainer and make changes to existing container when pulled by app, could we
change it in-place which can benefit: A. no need to create a new RMContainer,
B. no need to add them to "outstanding decrease/demote requests", C. pull new
demoted/decreased containers logics can be simplified in
SchedulerApplicationAttempt.
4) Not critical but worth to think: for increase/promote container, it increase
#live-containers by 1 in scheduler and soon decrease it. Since #live-containers
is visible to client, if changes to fix the issue is not too complex, it's
better to optimize this.
*Minor:*
5) SchedulerApplicationAttempt:
- IIUC, createResourceToShed is to assigned to the new RMContainer placeholder
to make scheduler can correctly release resources. Could you add some comments
and rename it to "getResourceOfRMContainerPlaceholder"
- A couple of rename suggestions to {{swapContainer}}: {{updatedContainer}} to
{{tempContainer}}; {{finalResource}} to {{updatedResource}}; {{newContainer}}
to {{updatedContainer}}; {{rmContainer}} to {{tempRmContainer}}
- Typo: updatType -> updateType
6) ContainerAllocator:
Not necessary to call regularContainerAllocator.assignContainers twice, once
should be enough. And not necessary to check reservedContainer anymore.
7) Not caused by your patch, testOrderOfIncreaseContainerRequestAllocation and
testIncreaseContainerRequestGetPreferrence are completely identical, you can
remove one of them.
8) Several unused methods in AppSchedulingInfo:
- hasRequestLabelChanged/getFirstSchedulingPlacementSet
> Unify Container Resizing code paths with Container Updates making it
> scheduler agnostic
> ---
>
> Key: YARN-6216
> URL: https://issues.apache.org/jira/browse/YARN-6216
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6216.001.patch, YARN-6216.002.patch
>
>
> YARN-5959 introduced an {{ContainerUpdateContext}} which can be used to
> update the ExecutionType of a container in a scheduler agnostic manner. As
> mentioned in that JIRA, extending that to encompass Container resizing is
> trivial.
> This JIRA proposes to remove all the CapacityScheduler specific code paths.
> (CapacityScheduler, CSQueue, FicaSchedulerApp etc.) and modify the code to
> use the framework introduced in YARN-5959 without any loss of functionality.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6222) TestFairScheduler.testReservationMetrics is flaky
[
https://issues.apache.org/jira/browse/YARN-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881057#comment-15881057
]
Yufei Gu commented on YARN-6222:
{{app.getAppAttemptResourceUsage().getPending();}} doesn't need update() in
FairScheduler to get latest pending demand, but {{app.getPendingDemand()}} does.
> TestFairScheduler.testReservationMetrics is flaky
> -
>
> Key: YARN-6222
> URL: https://issues.apache.org/jira/browse/YARN-6222
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6222.001.patch, YARN-6222.002.patch
>
>
> TestFairScheduler.testReservationMetrics is flaky because of timing issue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881025#comment-15881025 ] Varun Saxena commented on YARN-6027: Thanks [~rohithsharma] for the patch. As discussed offline, I am in agreement with generating fromId from storage layer as it can be different depending on storage layer. This is unlike UID which was primarily generated to replace multiple path params in REST URL and hence was storage independent. Also as in HBase from ID will be equivalent to row key, I am in agreement with generating it from RowKey classes. Few comments. # As from Id will have to be provided for most of the row keys, should we have another interface to encode and decode escaped row keys in String format and implement it in respective row keys converters along with KeyConverter interface? Thoughts? # Move decode(String fromId) and getRowKeyAsString methods to FlowActivityRowKeyConverter if above point is agreeable? And then rename these methods suitably in FlowActivityRowKey. # FlowActivityRowKey l.20: We have an unused import of java.util.Collections # FlowActivityRowKey l.119. IllegalArgumentException should be thrown. Also no need of appending result of getMessage IMO as its quite generic. # Can you add javadoc for fromId? # In TimelineReaderUtils, DEFAULT_DELIMITER_CHAR and DEFAULT_SEPARATOR_CHAR can be private. # We do not really have a public class for constants, so can we add FROMID_KEY as a public static constant in TimelineReaderUtils so that TestTimelineReaderWebServicesHBaseStorage#testGetFlowsForPagination can also use it. Just to avoid changing the test case if FROMID_KEY changes in future. # Nit: In the test case, &&fromid should be &fromid. # Can we have a couple of cases in test where enconding/decoding of separator and escape chars would be required? > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch, > YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, > YARN-6027-YARN-5355.0004.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6222) TestFairScheduler.testReservationMetrics is flaky
[ https://issues.apache.org/jira/browse/YARN-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880965#comment-15880965 ] Karthik Kambatla commented on YARN-6222: [~yufeigu] - do you want to share any stats from your flakiness tests? > TestFairScheduler.testReservationMetrics is flaky > - > > Key: YARN-6222 > URL: https://issues.apache.org/jira/browse/YARN-6222 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6222.001.patch, YARN-6222.002.patch > > > TestFairScheduler.testReservationMetrics is flaky because of timing issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880961#comment-15880961
]
Karthik Kambatla commented on YARN-6215:
Instead, would re-adding the app to the queue if we haven't preempted enough
containers be sufficient?
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
> Attachments: YARN-6215.001.patch
>
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880958#comment-15880958
]
Karthik Kambatla commented on YARN-6215:
Let me see if I understand this right.
bq. At this moment, a few queues has been updated while others are not. And the
preemptionThread will try to find container to preempt in the incomplete state.
Is this because the fairshares of other queues might not be updated by then?
And, we might not find enough containers to preempt in that round.
I am a little wary of adding a readlock on the scheduler. While that might
solve this problem, it increases the lock contention.
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
> Attachments: YARN-6215.001.patch
>
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6216) Unify Container Resizing code paths with Container Updates making it scheduler agnostic
[
https://issues.apache.org/jira/browse/YARN-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880946#comment-15880946
]
Hadoop QA commented on YARN-6216:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 0 new + 897 unchanged - 11 fixed = 897 total (was 908) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6216 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854232/YARN-6216.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 459412ddc0f3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959-branch-2.001.patch Attaching patch for branch-2 > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5959.005.patch, YARN-5959-branch-2.001.patch, > YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.003.patch, YARN-5959.wip.patch, YARN-5959-YARN-5085.001.patch, > YARN-5959-YARN-5085.002.patch, YARN-5959-YARN-5085.003.patch, > YARN-5959-YARN-5085.004.patch, YARN-5959-YARN-5085.005.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar to YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reopened YARN-5959: --- Re-openeing to include patch for branch-2 > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5959.005.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch, > YARN-5959-YARN-5085.001.patch, YARN-5959-YARN-5085.002.patch, > YARN-5959-YARN-5085.003.patch, YARN-5959-YARN-5085.004.patch, > YARN-5959-YARN-5085.005.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar to YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Description: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Demotion* : GUARANTEED to OPPORTUNISTIC. This is similar to YARN-1197 which allows for change in Container resources. was: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Demotion* : GUARANTEED to OPPORTUNISTIC. This is similar in YARN-1197 which allows for change in Container resources. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5959.005.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch, > YARN-5959-YARN-5085.001.patch, YARN-5959-YARN-5085.002.patch, > YARN-5959-YARN-5085.003.patch, YARN-5959-YARN-5085.004.patch, > YARN-5959-YARN-5085.005.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar to YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-1047) Expose # of pre-emptions as a queue counter
[ https://issues.apache.org/jira/browse/YARN-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1047: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-5990 > Expose # of pre-emptions as a queue counter > --- > > Key: YARN-1047 > URL: https://issues.apache.org/jira/browse/YARN-1047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.0.2-alpha >Reporter: Philip Zeyliger >Assignee: Karthik Kambatla > > Since YARN supports pre-empting containers, a given queue should expose the > number of containers it has had pre-empted as a metric. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-1047) Expose # of pre-emptions as a queue counter
[ https://issues.apache.org/jira/browse/YARN-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned YARN-1047: -- Assignee: Karthik Kambatla Target Version/s: 2.9.0 Component/s: fairscheduler > Expose # of pre-emptions as a queue counter > --- > > Key: YARN-1047 > URL: https://issues.apache.org/jira/browse/YARN-1047 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.0.2-alpha >Reporter: Philip Zeyliger >Assignee: Karthik Kambatla > > Since YARN supports pre-empting containers, a given queue should expose the > number of containers it has had pre-empted as a metric. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880886#comment-15880886
]
Tao Jie commented on YARN-6215:
---
[~kasha] [~yufeigu] [~sunilg], I uploaded a patch that add a readlock on
FsPreemptionThread. Would you mind giving it a review?
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
> Attachments: YARN-6215.001.patch
>
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Jie updated YARN-6215:
--
Attachment: YARN-6215.001.patch
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
> Attachments: YARN-6215.001.patch
>
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880872#comment-15880872
]
Tao Jie commented on YARN-6215:
---
I debugged this unittest and found out it is because of reentry of updateThread
and preemptionThread.
In updateThread, it goes through all queues and trigger the preemptionThread
once it finds a app is starved. At this moment, a few queues has been updated
while others are not. And the preemptionThread will try to find container to
preempt in the incomplete state.
Today updateThread is under writeLock of Fairscheduler, as a result, we need
to add a readLock of FS on the preemptionThread at the same time.
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880860#comment-15880860 ] Sangjin Lee commented on YARN-6027: --- Thanks [~rohithsharma] for the updated patch. I'll review it today. > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch, > YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, > YARN-6027-YARN-5355.0004.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Jie reassigned YARN-6215:
-
Assignee: Tao Jie
> TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in
> trunk
>
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6143) Fix incompatible issue caused by YARN-3583
[
https://issues.apache.org/jira/browse/YARN-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880705#comment-15880705
]
Hadoop QA commented on YARN-6143:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
37s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
30s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
19s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
43s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
26s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} | {color:orange} root: The patch generated 11 new + 157 unchanged
- 8 fixed = 168 total (was 165) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
48s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 37s{color}
[jira] [Commented] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880665#comment-15880665
]
Hadoop QA commented on YARN-5956:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5956 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854220/YARN-5956.10.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b77627a948aa 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a207aa9 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15059/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15059/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15059/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor ClientRMService
>
>
>
[jira] [Updated] (YARN-6216) Unify Container Resizing code paths with Container Updates making it scheduler agnostic
[
https://issues.apache.org/jira/browse/YARN-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6216:
--
Attachment: YARN-6216.002.patch
Updated patch (v002):
Fixing failed test case and javadoc
> Unify Container Resizing code paths with Container Updates making it
> scheduler agnostic
> ---
>
> Key: YARN-6216
> URL: https://issues.apache.org/jira/browse/YARN-6216
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6216.001.patch, YARN-6216.002.patch
>
>
> YARN-5959 introduced an {{ContainerUpdateContext}} which can be used to
> update the ExecutionType of a container in a scheduler agnostic manner. As
> mentioned in that JIRA, extending that to encompass Container resizing is
> trivial.
> This JIRA proposes to remove all the CapacityScheduler specific code paths.
> (CapacityScheduler, CSQueue, FicaSchedulerApp etc.) and modify the code to
> use the framework introduced in YARN-5959 without any loss of functionality.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6223) [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation on YARN
[ https://issues.apache.org/jira/browse/YARN-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880655#comment-15880655 ] Lei Guo commented on YARN-6223: --- [~leftnoteasy], based on the description "YARN scheduler should account GPU as a resource type just like CPU and memory", seems we are planning to consider the number of GPU as the resource. Depending on the GPU type, driver and toolkit, the sharing of one GPU among multiple applications could be important. During some discussion with domain expert, they do need the sharing of single GPU among applications, and do see the context switch overhead during the sharing. If the scheduling can let application know which core/memory to use, they can avoid some level context switch. We may need consider the scheduling of next level resource in GPU, at least not blocking future extension for next level resource scheduling. This is also related to the isolation part > [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation > on YARN > > > Key: YARN-6223 > URL: https://issues.apache.org/jira/browse/YARN-6223 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > > As varieties of workloads are moving to YARN, including machine learning / > deep learning which can speed up by leveraging GPU computation power. > Workloads should be able to request GPU from YARN as simple as CPU and memory. > *To make a complete GPU story, we should support following pieces:* > 1) GPU discovery/configuration: Admin can either config GPU resources and > architectures on each node, or more advanced, NodeManager can automatically > discover GPU resources and architectures and report to ResourceManager > 2) GPU scheduling: YARN scheduler should account GPU as a resource type just > like CPU and memory. > 3) GPU isolation/monitoring: once launch a task with GPU resources, > NodeManager should properly isolate and monitor task's resource usage. > For #2, YARN-3926 can support it natively. For #3, YARN-3611 has introduced > an extensible framework to support isolation for different resource types and > different runtimes. > *Related JIRAs:* > There're a couple of JIRAs (YARN-4122/YARN-5517) filed with similar goals but > different solutions: > For scheduling: > - YARN-4122/YARN-5517 are all adding a new GPU resource type to Resource > protocol instead of leveraging YARN-3926. > For isolation: > - And YARN-4122 proposed to use CGroups to do isolation which cannot solve > the problem listed at > https://github.com/NVIDIA/nvidia-docker/wiki/GPU-isolation#challenges such as > minor device number mapping; load nvidia_uvm module; mismatch of CUDA/driver > versions, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated YARN-5956:
-
Attachment: YARN-5956.10.patch
> Refactor ClientRMService
>
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch,
> YARN-5956.03.patch, YARN-5956.04.patch, YARN-5956.05.patch,
> YARN-5956.06.patch, YARN-5956.07.patch, YARN-5956.08.patch,
> YARN-5956.09.patch, YARN-5956.10.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880419#comment-15880419 ] Varun Saxena commented on YARN-6027: At a broad level, I am fine with the approach of doing FROM_ID encoding in row key classes. Will look at the patch in detail later. > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch, > YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, > YARN-6027-YARN-5355.0004.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880321#comment-15880321
]
Hadoop QA commented on YARN-6027:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
58s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 36 new
+ 0 unchanged - 0 fixed = 36 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
34s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 18s{co
[jira] [Commented] (YARN-5496) Make Node Heatmap Chart categories clickable
[
https://issues.apache.org/jira/browse/YARN-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880295#comment-15880295
]
Hadoop QA commented on YARN-5496:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5496 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854181/YARN-5496.001.patch |
| Optional Tests | asflicense |
| uname | Linux de054e832952 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a207aa9 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15057/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Make Node Heatmap Chart categories clickable
>
>
> Key: YARN-5496
> URL: https://issues.apache.org/jira/browse/YARN-5496
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Yesha Vora
>Assignee: Gergely Novák
> Attachments: YARN-5496.001.patch
>
>
> Make Node Heatmap Chart categories clickable.
> This Heatmap chart has few categories like 10% used, 30% used etc.
> This tags should be clickable. If user clicks on 10% used tag, it should show
> hosts with 10% usage. This can be a useful feature for clusters having 1000s
> of nodes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5496) Make Node Heatmap Chart categories clickable
[ https://issues.apache.org/jira/browse/YARN-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-5496: Attachment: YARN-5496.001.patch > Make Node Heatmap Chart categories clickable > > > Key: YARN-5496 > URL: https://issues.apache.org/jira/browse/YARN-5496 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Gergely Novák > Attachments: YARN-5496.001.patch > > > Make Node Heatmap Chart categories clickable. > This Heatmap chart has few categories like 10% used, 30% used etc. > This tags should be clickable. If user clicks on 10% used tag, it should show > hosts with 10% usage. This can be a useful feature for clusters having 1000s > of nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5496) Make Node Heatmap Chart categories clickable
[ https://issues.apache.org/jira/browse/YARN-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-5496: Attachment: (was: YARN-5496.001.patch) > Make Node Heatmap Chart categories clickable > > > Key: YARN-5496 > URL: https://issues.apache.org/jira/browse/YARN-5496 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Gergely Novák > Attachments: YARN-5496.001.patch > > > Make Node Heatmap Chart categories clickable. > This Heatmap chart has few categories like 10% used, 30% used etc. > This tags should be clickable. If user clicks on 10% used tag, it should show > hosts with 10% usage. This can be a useful feature for clusters having 1000s > of nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
