phet commented on code in PR #4047:
URL: https://github.com/apache/gobblin/pull/4047#discussion_r1752956735
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/workflow/impl/ExecuteGobblinWorkflowImpl.java:
##########
@@ -123,4 +170,44 @@ protected static WUProcessingSpec
createProcessingSpec(Properties jobProps, Even
}
return wuSpec;
}
+
+ private void cleanupWorkDirs(WUProcessingSpec workSpec,
EventSubmitterContext eventSubmitterContext, Set<String> directoriesToClean)
+ throws IOException {
+ // TODO: Add configuration to support cleaning up historical work dirs
from same job name
+ FileSystem fs = Help.loadFileSystem(workSpec);
+ JobState jobState = Help.loadJobState(workSpec, fs);
+ // TODO: Avoid cleaning up if work is being checkpointed e.g. midway of a
commit for EXACTLY_ONCE
+
+ if (PropertiesUtils.getPropAsBoolean(jobState.getProperties(),
ConfigurationKeys.CLEANUP_STAGING_DATA_BY_INITIALIZER, "false")) {
+ log.info("Skipping cleanup of work dirs for job due to initializer
handling the cleanup");
+ return;
+ }
+
+ DirDeletionResult dirDeletionResult =
deleteWorkDirsActivityStub.delete(workSpec, eventSubmitterContext,
+ calculateWorkDirsToDelete(jobState.getJobId(), directoriesToClean));
+
+ for (String dir : directoriesToClean) {
+ if (!dirDeletionResult.getSuccessesByDirPath().get(dir)) {
+ throw new IOException("Unable to delete one of more directories in the
DeleteWorkDirsActivity. Please clean up manually.");
Review Comment:
how about? -
```
List<String> failures = directoriesToClean.stream().filter(dir ->
!dirDeletionResult.getSuccessesByDirPath().get(dir)).collect(Collectors.toList);
if (!failures.isEmpty()) {
throw new IOException("Unable to delete work dirs: " + failures + ".
Please clean up manually.");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
