[
https://issues.apache.org/jira/browse/GOBBLIN-2152?focusedWorklogId=934067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-934067
]
ASF GitHub Bot logged work on GOBBLIN-2152:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 10/Sep/24 17:53
Start Date: 10/Sep/24 17:53
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #4047:
URL: https://github.com/apache/gobblin/pull/4047#discussion_r1752426178
##########
gobblin-temporal/src/test/java/org/apache/gobblin/temporal/ddm/workflow/impl/ExecuteGobblinWorkflowImplTest.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.workflow.impl;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+public class ExecuteGobblinWorkflowImplTest {
+
+ @Test
+ public void testCalculateWorkDirsDeletion() {
+ String jobId = "jobId";
+ Set<String> dirsToDelete = new HashSet<>();
+ dirsToDelete.add("/tmp/jobId/task-staging/");
+ dirsToDelete.add("/tmp/jobId/task-output/");
+ dirsToDelete.add("/tmp/jobId/task-output/file");
+ dirsToDelete.add("/tmp/jobId");
+ dirsToDelete.add("/tmp");
+ Set<String> result =
ExecuteGobblinWorkflowImpl.calculateWorkDirsToDelete(jobId, dirsToDelete);
+ Assert.assertEquals(result.size(), 4);
+ Assert.assertTrue(result.contains("/tmp/jobId/task-output/file"));
+ Assert.assertTrue(result.contains("/tmp/jobId/task-output"));
+ Assert.assertTrue(result.contains("/tmp/jobId/task-staging"));
+ Assert.assertTrue(result.contains("/tmp/jobId"));
Review Comment:
I changed the test so now I'll validate each folder directly, since the
result set also adds parents of folders
Issue Time Tracking
-------------------
Worklog Id: (was: 934067)
Time Spent: 20m (was: 10m)
> Gobblin Temporal Jobs should be properly cleaning up writer staging and
> output dirs
> -----------------------------------------------------------------------------------
>
> Key: GOBBLIN-2152
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2152
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: William Lo
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Gobblin HDFS writers write to a staging directory, which renames to an output
> directory, before committing data to the final destination. This is so that
> commits are atomic and not partially done (unless configured).
> Gobblin needs to clean up these directories in order to manage HDFS space
> properly so that failed jobs do not maintain their files around. It is
> critical to ensure that deletion is only done to folders that are configured
> at a job level, so that any folders that may be shared between jobs or
> concurrently are not deleted mid-execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
