[
https://issues.apache.org/jira/browse/TEZ-993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294938#comment-14294938
]
Jeff Zhang commented on TEZ-993:
--------------------------------
bq. whether to log recovery events for a DAG was not being done inside the
DAGImpl but in the recovery layer. Tomorrow, if we support a config that says a
particular DAG ( not a pre-warm dag ) should not be recovered, how would this
re-factor support this?
It should be easy to implement. Just need to add one flag in the first event:
DAGSubmittedEvent to tell RecoveryStore whether it is needed to recover. And
RecoveryStore would skipp the following events of this dag if the dag is not
required to recover.
bq. Apart from that, the recovery process needed to know the internals of the
DAGAppMaster such as dagCounter, etc. Currently, any relevant changes in the
DAGAppMaster will also need corresponding changes to the recovery restore layer.
In the patch, RecoveryStore has no reference of DAGAppMaster, and won't change
the state of DAGAppMaster explicitly. Instead, RecoveryStore wrap all the info
into RecoveredData, and pass it to DAGAppMaster, all the changes happen in
DAGAppMaster.
bq. what is the need for the Abstract class given that we have only one
implementation today?
This is for in case we will have other implementation just like yarn does for
RM recovery. (RMStateStore)
> Remove application logic from RecoveryService
> ---------------------------------------------
>
> Key: TEZ-993
> URL: https://issues.apache.org/jira/browse/TEZ-993
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Jeff Zhang
> Attachments: TEZ-993-3.patch, Tez-993-2.patch, Tez-993.patch
>
>
> Currently RecoveryService storage logic knows a lot about the DAG like which
> dag is pre-warm and does not need to be stored, which events needs special
> treatment etc. This kind of logic couples the DAG and the storage more than
> is probably necessary and can be a source of complications down the road. The
> storage should ideally be simply storing a sequence of arbitrary records
> delimited by a marker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
