[jira] [Assigned] (FLINK-11166) Slot Placement Constraint
[ https://issues.apache.org/jira/browse/FLINK-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger reassigned FLINK-11166: -- Assignee: (was: Abandoned Account) > Slot Placement Constraint > - > > Key: FLINK-11166 > URL: https://issues.apache.org/jira/browse/FLINK-11166 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Reporter: Abandoned Account >Priority: Major > > In many cases, users may want Flink to schedule their job tasks following > certain locality preferences. E.g., colocating upstream/downstream tasks to > reduce data transmission costs, dispersing tasks of certain pattern (e.g., > I/O intensive) to avoid resource competitions, running tasks in exclusive > TaskExecutor-s for task level resource consumption measurements, etc. > Currently, there are two ways in Flink to specify such locality preferences: > specifying preferred locations in the slot request, or setting slot sharing > group for the task. In both ways the preferences are specified when > requesting slots from the SlotPool and can affect how tasks are placed among > the slots allocated to the JobMaster. > However, there is no guarantee that such preferences can always be satisfied, > especially when slots are customized with different resource profiles for > different kinds of tasks. E.g., in cases where two tasks A and B are > preferred to be scheduled onto a same TaskExecutor, it is possible that none > of the slots customized for A offered to the JobMaster are collocated with > slots customized for B. > To better support locality preferences with various slot resource > specifications, it is necessary to allow JobMaster-s to request slots > subjected to certain placement constraints from the ResourceManager. > In addition, most underlying frameworks Flink runs on (Yarn, Kubernetes, > Mesos) already have individual supports for container level placement > constraints. It is a great opportunity for Flink to leverage such underlying > supports and enable scheduling tasks with rich locality preferences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-11166) Slot Placement Constraint
[ https://issues.apache.org/jira/browse/FLINK-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Xintong Song reassigned FLINK-11166: - Assignee: Tony Xintong Song > Slot Placement Constraint > - > > Key: FLINK-11166 > URL: https://issues.apache.org/jira/browse/FLINK-11166 > Project: Flink > Issue Type: New Feature > Components: ResourceManager >Reporter: Tony Xintong Song >Assignee: Tony Xintong Song >Priority: Major > > In many cases, users may want Flink to schedule their job tasks following > certain locality preferences. E.g., colocating upstream/downstream tasks to > reduce data transmission costs, dispersing tasks of certain pattern (e.g., > I/O intensive) to avoid resource competitions, running tasks in exclusive > TaskExecutor-s for task level resource consumption measurements, etc. > Currently, there are two ways in Flink to specify such locality preferences: > specifying preferred locations in the slot request, or setting slot sharing > group for the task. In both ways the preferences are specified when > requesting slots from the SlotPool and can affect how tasks are placed among > the slots allocated to the JobMaster. > However, there is no guarantee that such preferences can always be satisfied, > especially when slots are customized with different resource profiles for > different kinds of tasks. E.g., in cases where two tasks A and B are > preferred to be scheduled onto a same TaskExecutor, it is possible that none > of the slots customized for A offered to the JobMaster are collocated with > slots customized for B. > To better support locality preferences with various slot resource > specifications, it is necessary to allow JobMaster-s to request slots > subjected to certain placement constraints from the ResourceManager. > In addition, most underlying frameworks Flink runs on (Yarn, Kubernetes, > Mesos) already have individual supports for container level placement > constraints. It is a great opportunity for Flink to leverage such underlying > supports and enable scheduling tasks with rich locality preferences. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
