[
https://issues.apache.org/jira/browse/TEZ-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991296#comment-13991296
]
Mohammad Kamrul Islam commented on TEZ-693:
-------------------------------------------
[~hitesh] thanks for the feedback.
Agree with you the recovery files comment.
On the client generated files, i was considering whoever creates the file, she
should clean up it.
But I can see your point of fire-and-go client could keep a lot of files
footprint.
The new proposed approach in DagAM:
1. first determine if this attempt will be the last attempt. It can be
determined by checking two conditions:
A. appAttemptID.getAttemptId() >= maxAppAttempts
B. Step A is false and DagAM 'state' is DAGAppMasterState.SUCCEEDED and no
more new attempt will be tried.
2. If either Step 1.A or Step 1.B is 'true' , we can call file.deleteOnExit()
of those files.
Note: All these checks will depends on a user-defined property that is 'true'
by default.
> Deletion of DAG specific data after DAG completion
> --------------------------------------------------
>
> Key: TEZ-693
> URL: https://issues.apache.org/jira/browse/TEZ-693
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Mohammad Kamrul Islam
>
> Currently the client uploads some dag specific data to a remote directory
> specified by the user. The burden is on the client to clean this data after
> the dag completes. The post dag completion code in the AM should be able to
> clean this custom uploaded data.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
