[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347097#comment-16347097
]
Hudson commented on YARN-6594:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13589 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13589/])
YARN-6594. [API] Introduce SchedulingRequest object. (Konstantinos (arun
suresh: rev b57e8bc3002a95d2f2f328554d792151cdc1120d)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceSizing.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/SchedulingRequestPBImpl.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceSizingPBImpl.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/SchedulingRequest.java
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Major
> Fix For: YARN-6592
>
> Attachments: YARN-6594-YARN-6592.002.patch, YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226226#comment-16226226
]
Konstantinos Karanasos commented on YARN-6594:
--
Thanks [~wangda]! And [~asuresh], [~jianhe], [~sunilg] for the feedback/reviews!
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: YARN-6592
>
> Attachments: YARN-6594-YARN-6592.002.patch, YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223804#comment-16223804
]
Wangda Tan commented on YARN-6594:
--
Thanks [~kkaranasos] for updating the JIRA.
+1, will commit the patch next Monday.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594-YARN-6592.002.patch, YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190611#comment-16190611
]
Hadoop QA commented on YARN-6594:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-6592 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
33s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
9s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
19s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} YARN-6592 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
24s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
5s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6594 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890252/YARN-6594-YARN-6592.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 2cc3fed451cf 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126623#comment-16126623
]
Konstantinos Karanasos commented on YARN-6594:
--
bq. I just checked the latest patch, beyond the stable api below, patch looks
good. For rest of the debates, I suggest to move it to future tasks and not
block the feature development, what do you guys think?
Hi [~leftnoteasy], I agree. I will fix the issues with the patch within the
next days, so that we can move forward.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124369#comment-16124369
]
Hadoop QA commented on YARN-6594:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 9s{color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
10s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
8s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6594 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868567/YAR
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124317#comment-16124317
]
Wangda Tan commented on YARN-6594:
--
[~jianhe]/[~kkaranasos]/[~asuresh],
I just checked the latest patch, beyond the stable api below, patch looks good.
{code}
174 @Public
175 @Stable
176 public SchedulingRequest build() {
177 return schedulingRequest;
178 }
{code}
For rest of the debates, I suggest to move it to future tasks and not block the
feature development, what do you guys think?
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113268#comment-16113268
]
Konstantinos Karanasos commented on YARN-6594:
--
I see the use case for container upgrades, etc. -- thanks [~jianhe].
I just had an offline discussion with [~chris.douglas] and
[~arun.sur...@gmail.com]. It seems a better idea at the moment to have a first
simpler working version, as things might turn out to be more complicated than
we think if we add tags as the key-value pairs. We can extend it later, if need
be.
bq. It just occurred to me that, the design doc didn't cover changing/add the
container allocation tags, was that discussed ?
I agree this is a valid use case. We touched upon it briefly when discussing,
but agreed to address it later (similar to the increase/update container).
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112097#comment-16112097
]
Jian He commented on YARN-6594:
---
bq. BTW when would an AM want to select containers with specific attributes
though? Like when you are a service and want to update specific containers only?
Yep, e.g. I want to upgrade containers with specific label annotated. Also,
since I want to annotate metaInfo (most importantly, container-name) to the
container, if it's just a bag of strings, I won't know which string is for
container-name etc. unless I build my own parsing at client side.
One other option is to have underlying proto implementation as a map, but user
facing java API keeps as set and does transformation underneath. In case
there's a such use-case for it to be a map, we can extend the user-facing java
API, but the proto can keep the same. FYI, this
[link|https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature]
shows some use-case about how kubernetes models as key/value pairs for
affinity/anti-affinity between containers.
In any case, I'm ok to keep as-is because the current API is also sufficient
for my use-case.
It just occurred to me that, the design doc didn't cover changing/add the
container allocation tags, was that discussed ? I think at least we need to
account for such requirement when implementing. I do think this as a valid
use-case in future.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111595#comment-16111595
]
Konstantinos Karanasos commented on YARN-6594:
--
Thanks for the explanation, [~jianhe].
I see why key-value pairs will be needed to select containers. BTW when would
an AM want to select containers with specific attributes though? Like when you
are a service and want to update specific containers only?
Like you say, we could change the container tags to be key-value pairs. I was
trying to think of use cases that this would be really needed and I didn't find
many. For example, we could have {{memory_critical=high}} instead of just
{{memory-critical}}. But then the constraint API would get too complicated,
because we would need to teach people that they can have constraints where only
the key matters (as in "I want allocations that are memory_critical, no matter
the value") and others that the value matters too.
Adding they key-values in YARN-6593 is not hard, but I would suggest to keep it
as is for the first version.
For this JIRA, we can make the tags to be key-values, but only if we think it
is useful.
Another problem, like you say, is that the current tags refer to allocations,
while it seems that your key-value attributes would refer to containers.
Let me know what you guys think.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110288#comment-16110288
]
Jian He commented on YARN-6594:
---
On a second thought, if you think a set is more suitable for placement
scheduling because it essentially has only one type, that is the PLACEMENT.
I'm also fine to keep as-is.
Essentially, the use-case is different, no need to be forced to be the same.
I'll pursue what I need separately.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110276#comment-16110276
]
Jian He commented on YARN-6594:
---
Copy some context from YARN-6593
bq. The use-case is to be able to select containers based on key and values,
say, I want to find my containers with version=v1, and env = test, and foo
!=bar and name=web1 or name = web2. These are not used for making scheduling
decisions. But for annotating metaInfo to the containers. We can still make it
work by searching the entire string as Arun said. But that's not explicit. The
AM, client, or even UI then needs to parse the string to extract the keys and
values.
To support this, I probably need to add a separate filed (map) for this, call
it containerTags. and then the allocationTag in this jira can probably be named
as placementTags.
The questions is that the allocationTag right now is modeled as set, do we need
to make it as a key/value pair to be consistent ? In any case, a map can
support whatever a set can support, but it's not true the other way around. so
I think it will be more flexible.
I will pursue what I need as containerTags in a separate jira, but thought the
API might be better consistent.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103986#comment-16103986
]
Arun Suresh commented on YARN-6594:
---
Given that YARN-6593 looks close to committing, Think we should move forward
with this.
[~leftnoteasy] / [~jianhe], this we should discuss further about the changes to
the AMRMClient in YARN-6619
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081745#comment-16081745
]
Arun Suresh commented on YARN-6594:
---
So the reason I feel retaining the {{ContainerRequest}} is a good idea is
because it is strictly a super-set of what can be achieved with a
ResourceRequest. Furthermore, the main reason why we need AMRMClient today is
because most clients do not know that they HAVE to create 3 ResourceRequests
(node + rack + ANY) for a single container. The AMRMClient#ContainerRequest
hides all that.
Don't you think it would be better if we retain the ContainerRequest as the
de-facto API to be used by clients to ask for a container ?
Since you can specify all the required nodes, racks and labels in a single
ContainerRequest.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081309#comment-16081309
]
Wangda Tan commented on YARN-6594:
--
[~arun.sur...@gmail.com]
Agree with most of your points. Regarding to ContainerRequest, I'm not sure we
should keep it if we decide to deprecate AMRMClient. Same as my above comment,
it's highly bound to MR-like semantics.
I think we should not remove any of the AMRMClient/ContainerRequest logics,
just deprecate them. We can add support in RM side to convert placement-request
/ container-request / resource-request to the unified representation.
Applications can use AMRMClient as-is if they don't want to move.
I didn't see any problems to support logics like AM re-registers in the new
impl.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081225#comment-16081225
]
Arun Suresh commented on YARN-6594:
---
I agree.. AMRMClient is starting to get very heavy.
But fundamentally, I guess the biggest reason for the code bloat is:
# AMRMClient has to handle transformation of a ContainerRequest to the
corresponding ResourceRequests (the node/rack and ANY reqs) - I currently don't
know how we can do this transparantly to the end application. I propose we make
ContainerRequest a first class YARN API and deprecate the ResourceRequest
object.
# On RM failover etc, the AMRMClient ensures that the AM re-registers and it
also ensures that outstanding requests are resent.
[~subru] and I were discussing issues related to failover in a federated setup
- and we were thinking the same thing: deprecate AMRMClient
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081197#comment-16081197
]
Wangda Tan commented on YARN-6594:
--
[~kkaranasos], chatted with [~jianhe] offline. I think Jian's concern is more
about how AM use the new feature:
Existing applications interact with YARN by leveraging AMRMClient, however
AMRMClient is mostly design to support MR-like workload, and its design has
many issues such as in-consistent resource-request table between server/client.
I think now it is a good opportunity to think about how to fix the last-mile
problem. Do you think we should leverage AMRMClient or we should create a
brand-new ApplicationMaster Java API? I personally prefer to create a new one,
to me maintenance of legacy code to support new features is much harder than
adding a new implementation.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069248#comment-16069248
]
Konstantinos Karanasos commented on YARN-6594:
--
Thanks [~sunilg] -- yep, I will make sure I add tests/examples of how to use
the new API.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069020#comment-16069020
]
Sunil G commented on YARN-6594:
---
Thanks for the good work here folks!
I am starting to look this now.
[~kkaranasos] At high level it will be really great if we can have some
examples to show how to use these constructs for few real use cases.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068989#comment-16068989
]
Jian He commented on YARN-6594:
---
Sounds good, if SchedulingRequestBuilder has a same method for taking the
resource amount and number of containers, then it would be the same.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068874#comment-16068874
]
Konstantinos Karanasos commented on YARN-6594:
--
Hi [~jianhe], glad to hear you will be using the new API.
Re: your question on nesting the ResourceSizing object, I did it this way to
align with what we discussed offline with
[~asuresh]/[~curino]/[~chris.douglas]/[~subru]/[~vinodkv]/[~leftnoteasy].
Essentially, it is cleaner to separate the sizing object. Also we might need to
add more fields to it later, which will end up making the SchedulingRequest
object too bulky.
To make it easier for the applications, like you point out, I will make sure in
the next version I add a constructor in the SchedulingRequest that
automatically creates the ResourceSizing objects, given a number of allocations
and resources.
Hope it makes sense.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067273#comment-16067273
]
Jian He commented on YARN-6594:
---
got a question, any reason we add one more level abstraction for
{{ResourceSizing}} ?
If there's no logic implication behind it and this only serves as a simple
wrapper class, may be the existing flattened way is fine ? One less
abstraction layer in the API is easier to interact for the caller.
Btw, I'm going to use the APIs as soon as it's ready for the yarn native
services.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015037#comment-16015037
]
Konstantinos Karanasos commented on YARN-6594:
--
Cool, thanks [~leftnoteasy].
Will fix the @Stable. Also need to add the tests in the PBImpl that you pointed
out in YARN-6593 too.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015021#comment-16015021
]
Wangda Tan commented on YARN-6594:
--
Thanks [~kkaranasos], from a rough look, generally looks good to me.
Few comments:
1) There's a {{@Stable}} method, is it better to change to unstable since all
others are unstable?
2) ExecutionType is added to SchedulingRequest, I just not sure if it is a best
place to add. Let's keep it now and revisit it in the future.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
