[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116069#comment-14116069
]
Hadoop QA commented on YARN-1710:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665487/YARN-1712.1.patch
against trunk revision 9ad413b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4781//console
This message is automatically generated.
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.1.patch, YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129038#comment-14129038
]
Chris Douglas commented on YARN-1710:
-
{{GreedyReservationAgent}}
* Consider {{@link}} for {{ReservationRequest}} in class javadoc
* An inline comment could replace the {{adjustContract()}} method
* Most of the javadoc on private methods can be cut
* {{currentReservationStage}} does not need to be declared outside the loop
* {{allocations}} cannot be null
* An internal {{Resource(0, 0)}} could be reused
* {{li}} should be part of the loop ({{for}} not {{while}}). Its initialization
is unreadable; please use temp vars.
* Generally, embedded calls are difficult to read:
{code}
if (findEarliestTime(allocations.keySet()) > earliestStart) {
allocations.put(new ReservationInterval(earliestStart,
findEarliestTime(allocations.keySet())), ReservationRequest
.newInstance(Resource.newInstance(0, 0), 0));
// consider to add trailing zeros at the end for simmetry
}
{code}
Assuming the {{ReservationRequest}} is never modified by the plan:
{code}
private final ZERO_RSRC =
ReservationRequest.newInstance(Resource.newInstance(0, 0), 0);
// ...
long allocStart = findEarliestTime(allocations.keySet());
if (allocStart > earliestStart) {
ReservationInterval preAlloc =
new ReservationInterval(earliestStart, allocStart);
allocations.put(preAlloc, ZERO_RSRC);
}
{code}
* {{findEarliestTime(allocations.keySet())}} is called several times and should
be memoized
** Would a {{TreeSet}} be more appropriate, given this access pattern?
* Instead of:
{code}
boolean result = false;
if (oldReservation != null) {
result = plan.updateReservation(capReservation);
} else {
result = plan.addReservation(capReservation);
}
return result;
{code}
Consider:
{code}
if (oldReservation != null) {
return plan.updateReservation(capReservation);
}
return plan.addReservation(capReservation);
{code}
* A comment unpacking the arithmetic for calculating {{curMaxGang}} would help
readability
{{TestGreedyReservationAgent}}
* Instead of fixing the seed, consider setting and logging it for each run.
* {{testStress}} is brittle, as it verifies only the timeout; {{testBig}} and
{{testSmall}} don't verify anything. Both tests are useful, but probably not as
part of the build. Dropping the annotation and adding a {{main()}} that calls
each fo them would be one alternative.
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.1.patch, YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131026#comment-14131026 ] Carlo Curino commented on YARN-1710: [~chris.douglas], thanks for the always insightful feedback. I attach a patch addressing all your comments with two minor exceptions: * I am not memoizing findEarliestTime, as it would only save one invocation (the others are on diff sets, or updated version of the same set). Similarly TreeSet would only help partially, as we need to search for both min(startTime) and max(endTime), hence no single ordering can help both. * for the iterator on stages, I find the for loop less readable than a while (as the initialization and increment will be large/not very readable) The rest is done as you suggest. Let me know if it looks good or if there is anything else to be addressed. > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131035#comment-14131035
]
Hadoop QA commented on YARN-1710:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668243/YARN-1710.2.patch
against trunk revision 6b8b160.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4909//console
This message is automatically generated.
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131700#comment-14131700
]
Chris Douglas commented on YARN-1710:
-
bq. I am not memoizing findEarliestTime, as it would only save one invocation
(the others are on diff sets, or updated version of the same set)
I'm confused. There are three invocations:
{code}
if (findEarliestTime(allocations.keySet()) > earliestStart) {
allocations.put(new ReservationInterval(earliestStart,
findEarliestTime(allocations.keySet())), ZERO_RES);
}
ReservationAllocation capReservation =
new InMemoryReservationAllocation(reservationId, contract, user,
plan.getQueueName(), findEarliestTime(allocations.keySet()),
findLatestTime(allocations.keySet()), allocations,
plan.getResourceCalculator(), plan.getMinimumAllocation());
{code}
Isn't earliest time is either the earliest in the set, or the interval this
just added?
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131772#comment-14131772 ] Carlo Curino commented on YARN-1710: I understand what you meant (after our brief chat). I addressed it in the version I just uploaded (v3). > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131797#comment-14131797
]
Hadoop QA commented on YARN-1710:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668391/YARN-1710.3.patch
against trunk revision 78b0483.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4926//console
This message is automatically generated.
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch,
> YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132356#comment-14132356 ] Carlo Curino commented on YARN-1710: Minor fix addressing [~chris.douglas] comments further. > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.4.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132383#comment-14132383
]
Hadoop QA commented on YARN-1710:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668512/YARN-1710.4.patch
against trunk revision a0ad975.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4949//console
This message is automatically generated.
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch,
> YARN-1710.4.patch, YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132466#comment-14132466 ] Chris Douglas commented on YARN-1710: - +1 lgtm. Thanks [~curino] for all the iterations on this > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.4.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158765#comment-14158765 ] Hudson commented on YARN-1710: -- FAILURE: Integrated in Hadoop-trunk-Commit #6189 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6189/]) YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev f66ffcf832235e0da0bb050fff08e248b547c360) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/GreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/ReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestGreedyReservationAgent.java * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.4.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159036#comment-14159036 ] Hudson commented on YARN-1710: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #700 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/700/]) YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev f66ffcf832235e0da0bb050fff08e248b547c360) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/GreedyReservationAgent.java * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestGreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/ReservationAgent.java > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.4.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159143#comment-14159143 ] Hudson commented on YARN-1710: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1891 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1891/]) YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev f66ffcf832235e0da0bb050fff08e248b547c360) * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestGreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/GreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/ReservationAgent.java > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.4.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159167#comment-14159167 ] Hudson commented on YARN-1710: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1916 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1916/]) YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev f66ffcf832235e0da0bb050fff08e248b547c360) * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/ReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/GreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestGreedyReservationAgent.java > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.1.patch, YARN-1710.2.patch, YARN-1710.3.patch, > YARN-1710.4.patch, YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044243#comment-14044243 ] Carlo Curino commented on YARN-1710: I uploaded a patch that contains a GreedyReservationAgent. The philosophy behind this agent is to greedily place from the deadline back towards arrival. This intuitively increase the chances of a job showing up later but with an earlier deadline to find space in the plan. It works well when paired with opportunistic anticipation of work (i.e., thanks to YARN-1957 even if the reservation has a zero allocation at the moment the scheduler can give it resources if they are not claim by anyone.). Smarter placement policies are worth thinking about, and will increase the % of jobs accepted, or minimize other parameters such as preemption. We are exploring this space. > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044353#comment-14044353
]
Hadoop QA commented on YARN-1710:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652531/YARN-1710.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4084//console
This message is automatically generated.
> Admission Control: agents to allocate reservation
> -
>
> Key: YARN-1710
> URL: https://issues.apache.org/jira/browse/YARN-1710
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-1710.patch
>
>
> This JIRA tracks the algorithms used to allocate a user ReservationRequest
> coming in from the new reservation API (YARN-1708), in the inventory
> subsystem (YARN-1709) maintaining the current plan for the cluster. The focus
> of this "agents" is to quickly find a solution for the set of contraints
> provided by the user, and the physical constraints of the plan.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044414#comment-14044414 ] Carlo Curino commented on YARN-1710: As discussed before the patches are not meant to be applicable independently. We are working on YARN-1051 dev branch for that. > Admission Control: agents to allocate reservation > - > > Key: YARN-1710 > URL: https://issues.apache.org/jira/browse/YARN-1710 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1710.patch > > > This JIRA tracks the algorithms used to allocate a user ReservationRequest > coming in from the new reservation API (YARN-1708), in the inventory > subsystem (YARN-1709) maintaining the current plan for the cluster. The focus > of this "agents" is to quickly find a solution for the set of contraints > provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.2#6252)
