[
https://issues.apache.org/jira/browse/TEZ-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219187#comment-14219187
]
Jeff Zhang commented on TEZ-992:
--------------------------------
Seems this jira related to TEZ-714 closely, I try to resolve them together. I
create a FinishSavingService for logging recovery event and commit/abort.
Attach the 2 state machines diagrams (DAG/Vertex)
* The main change is that I add one additional state: FINISH_SAVING. (
RUNNING/TERMINATING will transite to FINISH_SAVING first and then go to
SUCCEEDED/FAILED/KILLED )
** In Vertex's FINISH_SAVING, it will log recovery data (VertexFinisheEvent,
VertexCommitStartedEvent) and commit/abort the data if necessary.
** in DAG's FINISH_SAVING, it will log recovery data (DAGCommitStartedEvent,
DAGFinishedEvent) and commit the/abort data if necessary
** For VertexGroupCommitStartedEvent / VertexGroupCommitFinishedEvent, I will
run it in FinishSavingService and keep DAG in the state of RUNNING.
** In recovery, still keep logging recovery event and commit in
AsycMainDispatcher. Not so confidently on moving it out now, may leave it in
another jira.
I have implemented a prototype of this feature ( run tez examples successfully
and TestAMRecovery successfully ) but still need some code refinement, will
attach the patch soon.
When I implement it, I still have the following questions, hope to get some
feedback about them.
* initializing of committer is still in the main AsycDispather thread, is it
acceptable ?
* Can TerminateEvent been ignored when DAG/Vertex is in FINISH_SAVING ? IMO, I
think it can been ignored. Because if we don't ignore it, we still need to
abort the committer. Since for both ignore and not-ingore we both have to call
commit or abort, I think ignore it is acceptable.
> Recovery data should not be written on AsyncDispatcher thread
> -------------------------------------------------------------
>
> Key: TEZ-992
> URL: https://issues.apache.org/jira/browse/TEZ-992
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Jeff Zhang
>
> This may block the DAG operations in case the recovery data needs to be
> synchronously stored. The operations requiring this blocking operation should
> change their state machines to wait for the store operation before moving
> ahead. They will move ahead after they receive notification from the
> RecoveryService that their operation has completed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
