zentol opened a new pull request #13508:
URL: https://github.com/apache/flink/pull/13508
Adds the `SlotTracker`, a component of the declarative slot manager that
tracks task executor slots.
This PR contains a few preparatory commits:
- The (Slot)State is now a top-level class, because we are about to add a
new kind of slot data structure (trimmed down version of TaskManagerSlot), and
it would be good to re-use this enum.
- Add a default implementation for
`TaskManagerSlotInformation#isMatchingRequirement`; this method is fairly
uninteresting and no implementation should require a different one. Mostly
added so we don't have to duplicate code for the new slot implementation.
- Add `TaskExecutorConnection` to `TaskManagerSlotInformation`: The
`TaskManagerSlotInformation` is an abstraction over slots, but was lacking a
crucial piece to allow the slot manager to operate solely on these information
objects. With the `TaskExecutorConnection` added to the interface the slot
implementations can now be a fully hidden from the slot manager.
Finally, onto the main commit:
`DeclarativeTaskManagerSlot`: Contains the information necessary for
allocating the slot, and a state machine. It is a heavily trimmed down version
of the `TaskManagerSlot` for the declarative resource management. Most notable
it is not aware of pending slots (which leaks details for how task executors
are allocated) nor allocation futures (which leaks details for how slots are
assigned and how the RPC system works).
Furthermore it enforces a strict slot life-cycle:
```
FREE -> Pending -> Allocated -> FREE
|> FREE
```
`SlotStatusUpdateListener`: A listener interface for slot status
notifications, send out by the `SlotTracker`. This only includes changes in
slot states; there are no notifications for added/removed slots (due to there
being no need for it). This interface will be used by the slot manager to keep
multiple data-structures up to date, including (but not limited to) the
ResourceTracker (FLINK-19307) and TaskExecutorAllocater(FLINK-19309).
`SlotTracker`: Tracks slots and their state, the main purposes being to tell
which slots are free and informing other components about status updates. It is
informed by the slot manager whenever slots are added/removed or when the state
of a slot changes, be it either due to the slot manager initiating slot
allocations or the task executor reporting about the slot statuses. This
component is generally not aware of task executors however, essentially acting
as a pool of slots.
There are 2 code paths for status updates:
1) the slot manager executes state transitions when
starting/finishing/cancelling slot allocated. These transitions must follow the
strict life-cycle of the `DeclarativeTaskManagerSlot` (any failure to do is is
likely a bug)
2) the task executor report the status of all slots, which are treated as
the source of truth. On a high-level these transitions may be illegal (e.g., a
slot tracked as FREE being reported as allocated). Such transitions reconciled
with the currently tracked state by the `SlotStatusStateReconciler` in the
`DefaultSlotTracker`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
