Yun Gao created FLINK-12865:
-------------------------------
Summary: State inconsistency between RM and TM on the slot status
Key: FLINK-12865
URL: https://issues.apache.org/jira/browse/FLINK-12865
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Reporter: Yun Gao
Assignee: Yun Gao
There may be state inconsistency between TM and RM due to race condition and
message loss:
# When TM sends heartbeat, it retrieve SlotReport in the main thread, but
sends the heartbeat in another thread. There may be cases that the slot on TM
is FREE initially and SlotReport read the FREE state, then RM requests slot and
mark the slot as allocated, and the SlotReport finally override the allocated
status at the RM side wrongly.
# When RM requests slot, TM received the requests but the acknowledge message
get lot. Then RM will think this slot is free and assigned it to other request.
Add a version to the slot may solves this problem.
# Add a SYNCING status in the RM side. SYNCING means request is sent to TM but
not knowing the result of the request. A slot with SYNCING status cannot be
allocated to others.
# RM and TM maintains a version for each slot, and the version starts from 0.
# Whenever RM requests slot, it add the version by 1, and send the requests to
TM. TM will only do allocation when RM's version > TM's version.
# TM will also attach the version in the HeartBeat and RM will only accept the
slot status when the TM's version >= RM's version.
The version method is a simplified solution of the full vector clock based
state management. In the full vector-clock design, the version should be a
vector represents (RM's version, TM's version). Whenever RM modify the slot's
status (requestSlot) and TM modify the slot's status (freeSlot), It need to
first increase the corresponding component and send the sync messages, and the
messages can only be accepted when the vector version >= messages' vector
version.
However, since for the status of slot TM will only modify its status when
freeing slots, we can ignore the component of TM's side will only cause a freed
slot be marked as allocated, this will not cause error, and the free status can
be finally updated to RM with Heartbeat message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
