[
https://issues.apache.org/jira/browse/TEZ-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113433#comment-14113433
]
Jeff Zhang edited comment on TEZ-853 at 8/28/14 7:06 AM:
---------------------------------------------------------
[~hitesh]
bq. Do TaskImpl and VertexImpl write counters to recovery but they are not used
when restoring state? Should the counters be written or recovered from task
attempts? If the latter, then we should not write them.
There's no counter written from VertexImpl and TaskImpl ( their counters are
from TaskAttemptImpl ) Should we remove tezCounters from VertexFinishedProto
and TaskFinishedProto ? since actually we don't use it in recovery
bq. DAGImpl::restoreFromEvent does not seem to restore counters
DAG do not write any counters, its counters are all from TaskAttemptImpl. So
that means as long as counters of TaskAttemptImpl is recovered, Counters of DAG
is recovered.
bq. in a scenario where the dag finished is logged and all other events are
dropped, I assume counters will be needed?
Yes, you are right.This is a special case. In this case we should write
counters in DAGFinishedEvent and recover from it. ( will add it )
was (Author: zjffdu):
[~hitesh]
bq. Do TaskImpl and VertexImpl write counters to recovery but they are not used
when restoring state? Should the counters be written or recovered from task
attempts? If the latter, then we should not write them.
There's no counter written from VertexImpl and TaskImpl ( their counters are
from TaskAttemptImpl )
bq. DAGImpl::restoreFromEvent does not seem to restore counters
DAG do not write any counters, its counters are all from TaskAttemptImpl. So
that means as long as counters of TaskAttemptImpl is recovered, Counters of DAG
is recovered.
bq. in a scenario where the dag finished is logged and all other events are
dropped, I assume counters will be needed?
Yes, you are right.This is a special case. In this case we should write
counters in DAGFinishedEvent and recover from it. ( will add it )
> Support counters recovery
> -------------------------
>
> Key: TEZ-853
> URL: https://issues.apache.org/jira/browse/TEZ-853
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Hitesh Shah
> Assignee: Jeff Zhang
> Attachments: Tez-853.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
