[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.07.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
> YARN-8827-YARN-1011.04.patch, YARN-8827-YARN-1011.05.patch,
> YARN-8827-YARN-1011.06.patch, YARN-8827-YARN-1011.07.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.06.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
> YARN-8827-YARN-1011.04.patch, YARN-8827-YARN-1011.05.patch,
> YARN-8827-YARN-1011.06.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.05.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
> YARN-8827-YARN-1011.04.patch, YARN-8827-YARN-1011.05.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.04.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
> YARN-8827-YARN-1011.04.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.03.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.02.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.01.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Description:
Opportunistic Containers for OverAllocation need to be allocated to pending
applications in some fair manner. Rather than evaluating queue and user
resource usage (allocated resource usage) and comparing against queue and user
limits to decide the allocation, it might make more sense to use a snapshot of
actual resource utilization of the queue and user.
To facilitate this, this JIRA proposes to aggregate per user, per app (and
maybe per queue) resource utilization in addition to aggregated Container and
Node Utilization and send it along with the NM heartbeat. It should be fairly
inexpensive to aggregate - since it can be performed in the same loop of the
{{ContainersMonitorImpl}}'s Monitoring thread.
A snapshot aggregate can be made every couple of seconds in the RM. This
instantaneous resource utilization should be used to decide if Opportunistic
containers can be allocated to an App, Queue or User.
was:
Opportunistic Containers for OverAllocation need to be allocated to pending
applications in some fair manner. Rather than evaluating queue and user
resource usage (allocated resource usage) and comparing against queue and user
limits to decide the allocation, it might be make more sense to use a snapshot
of actual resource utilization of the queue and user.
To facilitate this, this JIRA proposes to aggregate per user, per app (and
maybe per queue) resource utilization in addition to aggregated Container and
Node Utilization and send it along with the NM heartbeat. It should be fairly
inexpensive to aggregate - since it can be performed in the same loop of the
{{ContainersMonitorImpl}}'s Monitoring thread.
A snapshot aggregate can be made every couple of seconds in the RM. This
instantaneous resource utilization should be used to decide if Opportunistic
containers can be allocated to an App, Queue or User.
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
