[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347114#comment-16347114
]
Hudson commented on YARN-7774:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13589 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13589/])
YARN-7774. Miscellaneous fixes to the PlacementProcessor. (asuresh) (arun
suresh: rev 28fe7f331837b36e78fa34ed990993677dddeaee)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SingleConstraintAppPlacementAllocator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/TestSingleConstraintAppPlacementAllocator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/processor/BatchedRequests.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/algorithm/DefaultPlacementAlgorithm.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintsUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestPlacementProcessor.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/algorithm/CircularIterator.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/algorithm/TestCircularIterator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Fix For: YARN-6592
>
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333135#comment-16333135
]
Sunil G commented on YARN-7774:
---
Makes sense. Sorry, I missed that one. Could you also update the exception in
the same block, though maxCardinality check is != 0, exception message still
says != 1 in {{SingleConstraintAppPlacementAllocator}}. If any Jiras are abt to
commit, we can get this in or ll file a trivial one later.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Fix For: YARN-6592
>
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333132#comment-16333132
]
Arun Suresh commented on YARN-7774:
---
[~sunilg], so, do take a look at [this
comment|https://issues.apache.org/jira/browse/YARN-7774?focusedCommentId=16331286&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16331286].
As per that, the max cardinality is the cardinality the constraint satisfier
will see BEFORE the placing the container.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Fix For: YARN-6592
>
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333127#comment-16333127
]
Sunil G commented on YARN-7774:
---
A quick doubt. {{SingleConstraintAppPlacementAllocator}} why maxCardinality
validation is changed from 1 to 0?
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Fix For: YARN-6592
>
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333121#comment-16333121
]
Weiwei Yang commented on YARN-7774:
---
Hi [~asuresh], that sounds good to me. +1 on latest patch. I will try more
tests when this one gets in, and file JIRAs if I found any issue or improvement
can be made. Thank you.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333097#comment-16333097
]
Arun Suresh commented on YARN-7774:
---
The test case error is unrelated. [~Weiwei Yang] , [~kkaranasos] , can we move
forward with the latest patch? Happy to file subsequent JIRAs for anything
specific.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332945#comment-16332945
]
genericqa commented on YARN-7774:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} YARN-6592 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
55s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} YARN-6592 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 115 unchanged - 0 fixed = 117 total (was 115)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 38s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 59s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7774 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906882/YARN-7774-YARN-6592.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 655580a91755 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-6592 / 27fa101 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19354/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19354/artifact/out/patch-unit-hadoop-
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332847#comment-16332847
]
Arun Suresh commented on YARN-7774:
---
Updating patch
* Moved circular iterator to its own class
* The algorithm follows the scheme I mentioned in my previous comment. It will
try the last satisfied node before checking the next node in the iteration.
* Fixed the tests. The {{TestContinuousScheduling}} is unrelated and works fine
for me.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch, YARN-7774-YARN-6592.005.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332708#comment-16332708
]
Arun Suresh commented on YARN-7774:
---
Thanks for the comments [~kkaranasos] and [~cheersyang].
bq. CircularIterator looks to be general enough and deserves its own class with
a generic type, it can be moved to common package..
Will do - i had initially done that. Only issue was we need the element to
implement equals, which the SchedulerNode does not (I had to explicitly get the
nodeId from the schedulerNode and compare it :)) - But yeah, Ill move to a
different class, and that way I can put in some unit tests.
bq. it mean the second allocation can only be made after it iterates all nodes
again? This algorithm doesn't seem to be affinity friendly.
Yup, I did consider that. My thought process was: Given that anti-affinity will
probably be used more often, and even if affinity is used - we've come across
more use cases for RACK and label affinity, than node affinity - Both these
cases are served better with the CircularIterator.
bq. Do we really need the CircularIterator? It seems to me that you can have a
normal iterator initialized outside the for loop and then each time
hasNext()=false, you can re-initialize it. But maybe I am missing something
Yeah, its mostly equivalant (we also need to record the starting element), but
I did not want to clutter the the main loop of the code with additional
variables etc., abstracting out to a different class looked cleaner.
Regarding using the minCardinality > 0 check anti affinity:
Instead of looking at the mincardinalty, I was thinking the following scheme
would be efficient for both node-affinity as well as anti-affinity. If a
SchedulingRequest has been placed on a node, and if there are more Requests,
then instead of initializing the circularIterator from the next node, we
initialize it starting from the CURRENT node. That way, if the next request (or
even if the previous req has numAllocations > 1) the just-placed-node will be
the first candidate - It will also not severely impact anti-affinity placement,
since only a single previously considered node will be re-considered before
moving on to the rest of the cluster. Makes sense ?
bq. Do we clean up the black list for each tag? It seems that black-listing can
change based on the allocations that have been done so far, so we might need to
use it carefully.
The blackists are not persistent - they only hang around time the max retries
are completed - after which both the requests and the blacklists are discarded.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331969#comment-16331969
]
Konstantinos Karanasos commented on YARN-7774:
--
Thanks [~asuresh] for the patch. Looks good overall, some comments:
* Do we really need the CircularIterator? It seems to me that you can have a
normal iterator initialized outside the for loop and then each time
hasNext()=false, you can re-initialize it. But maybe I am missing something.
* For what [~cheersyang] mentioned about not being affinity friendly, you
could make a check whether the constraint has minCardinality>0 and scope=NODE,
and then keep the iterator at the same place in such a case. But if you feel it
is an over-optimization for the time being, I am fine tackling it in another
JIRA. Up to you.
* Do we clean up the black list for each tag? It seems that black-listing can
change based on the allocations that have been done so far, so we might need to
use it carefully.
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331868#comment-16331868
]
Weiwei Yang commented on YARN-7774:
---
Hi [~asuresh]
In \{{DefaultPlacementAlgorithm}}, for each SchedulingRequest, it iterates over
available nodes and each time it attempts to allocate one allocation on a node,
then go to next node. Imagine the request asks for 2 allocations affinity on
same node, does it mean the second allocation can only be made after it
iterates all nodes again? This algorithm doesn't seem to be affinity friendly.
{\{CircularIterator}} looks to be general enough and deserves its own class
with a generic type, it can be moved to common package. Also it would be good
to add some test case for it. It doesn't have to be done in this patch. Just a
suggestion.
Hope this helps. Thanks
> Miscellaneous fixes to the PlacementProcessor
> -
>
> Key: YARN-7774
> URL: https://issues.apache.org/jira/browse/YARN-7774
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Blocker
> Attachments: YARN-7774-YARN-6592.001.patch,
> YARN-7774-YARN-6592.002.patch, YARN-7774-YARN-6592.003.patch,
> YARN-7774-YARN-6592.004.patch
>
>
> JIRA to track the following minor changes:
> * Scheduler must normalize requests that are made using the
> {{attemptAllocationOnNode}} method.
> * Currently, the placement algorithm resets the node iterator for each
> request. The Placement Algorithm should either shuffle the node iterator OR
> use a circular iterator - to ensure a) more nodes are looked at and b) bias
> against placing too many containers on the same node
> * Add a placement retry loop for rejected requests - since there are cases
> especially, when Constraints will be satisfied after a subsequent request has
> been placed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331852#comment-16331852
]
genericqa commented on YARN-7774:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
57s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-6592 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
0s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} YARN-6592 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 52s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSchedulingRequestUpdate
|
| |
hadoop.yarn.server.resourcemanager.scheduler.placement.TestSingleConstraintAppPlacementAllocator
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7774 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906767/YARN-7774-YARN-6592.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 758dcb3d77cb 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-6592 / 27fa101 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19344/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
http
[jira] [Commented] (YARN-7774) Miscellaneous fixes to the PlacementProcessor
[
https://issues.apache.org/jira/browse/YARN-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331835#comment-16331835
]
genericqa commented on YARN-7774:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-6592 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
6s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} YARN-6592 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 109 unchanged - 0 fixed = 116 total (was 109)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 33s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSchedulingRequestUpdate
|
| |
hadoop.yarn.server.resourcemanager.scheduler.placement.TestSingleConstraintAppPlacementAllocator
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7774 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906766/YARN-7774-YARN-6592.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux aef862c32295 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-6592 / 27fa101 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19343/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-ya
