[
https://issues.apache.org/jira/browse/YARN-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304849#comment-15304849
]
Carlo Curino commented on YARN-5164:
------------------------------------
Thinking more about this, it is non-trivial to RLE-ify this policy. The policy
computes for every sliding window the integral of used resources and make sure
it does not exceed an integral quota. The issue with this is that even for a
constant value of the user reservation (or some of reservations) the computed
integral over a moving window changes.
In the attached *YARN-5164-example.pdf* we visualize this for a simple 5-sample
window. The orange line changes even for constant values of the blue line.
We could in principle leverage higher-order representations (where we store
RLE-encoded vectors for non-horizontal lines as starting-point + slope), but
this is far from trivial, and we need to make sure the code complexity is
justified. Postponing this for now.
> CapacityOvertimePolicy does not take advantaged of plan RLE
> -----------------------------------------------------------
>
> Key: YARN-5164
> URL: https://issues.apache.org/jira/browse/YARN-5164
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler, fairscheduler, resourcemanager
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: YARN-5164-example.pdf
>
>
> As a consequence small time granularities (e.g., 1 sec) and long time horizon
> for a reservation (e.g., months) run rather slow (10 sec).
> Proposed resolution is to switch to interval math in checking, similar to how
> YARN-4359 does for agents.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
