umustafi commented on code in PR #3899:
URL: https://github.com/apache/gobblin/pull/3899#discussion_r1543298328
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/DagTask.java:
##########
@@ -41,25 +42,33 @@
public abstract class DagTask {
@Getter public final DagActionStore.DagAction dagAction;
private final MultiActiveLeaseArbiter.LeaseObtainedStatus
leaseObtainedStatus;
+ private final DagActionStore dagActionStore;
@Getter protected final DagManager.DagId dagId;
- public DagTask(DagActionStore.DagAction dagAction,
MultiActiveLeaseArbiter.LeaseObtainedStatus leaseObtainedStatus) {
+ public DagTask(DagActionStore.DagAction dagAction,
MultiActiveLeaseArbiter.LeaseObtainedStatus leaseObtainedStatus,
+ DagActionStore dagActionStore) {
this.dagAction = dagAction;
this.leaseObtainedStatus = leaseObtainedStatus;
+ this.dagActionStore = dagActionStore;
this.dagId = DagManagerUtils.generateDagId(dagAction.getFlowGroup(),
dagAction.getFlowName(), dagAction.getFlowExecutionId());
}
public abstract <T> T host(DagTaskVisitor<T> visitor);
/**
- * Any cleanup work, e.g. releasing lease if it was acquired earlier, may be
done in this method.
+ * Any cleanup work, including removing the dagAction from the
dagActionStore and completing the lease acquired to
+ * work on this task, is done in this method.
* Returns true if concluding dag task finished successfully otherwise false.
*/
// todo call it from the right place
public boolean conclude() {
try {
+ // If dagAction is not present in store then it has been deleted by
another participant, so we can skip deletion
+ if (this.dagActionStore.exists(this.dagAction)) {
Review Comment:
I guess delete is already idempotent by nature of where clause `"DELETE FROM
%s WHERE flow_group = ? AND flow_name =? AND flow_execution_id = ? AND job_name
= ? AND dag_action = ?";` no need to check then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
