Myasuka commented on a change in pull request #7942: [FLINK-11696][checkpoint]
Avoid to send mkdir requests to DFS from task side
URL: https://github.com/apache/flink/pull/7942#discussion_r264354794
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorage.java
##########
@@ -107,6 +105,13 @@ public CheckpointStorageLocation
initializeLocationForCheckpoint(long checkpoint
// prepare all the paths needed for the checkpoints
final Path checkpointDir =
createCheckpointDirectory(checkpointsDirectory, checkpointId);
+ if (!areDirectoriesCreated) {
+ fileSystem.mkdirs(checkpointsDirectory);
+ fileSystem.mkdirs(sharedStateDirectory);
Review comment:
I think a new interface `ChekcpointCoordinatorStorgae` extending
`ChckecpointStorgae` with `#supportsHighlyAvailableStorage` ,
`#hasDefaultSavepointLocation`, `#resolveCheckpoint`,
`#initializeLocationForCheckpoint` and `#initializeLocationForSavepoint`
methods could be introduced, while previous `CheckpointStorage` would only have
two methods: `#resolveCheckpointStorageLocation` and
`#createTaskOwnedStateStream`.
In JM side, we could cast the created storage as
`ChekcpointCoordinatorStorgae` so that we could have those methods while TM
side would not call those methods accidentally.
I think this would be clearer for users to distinguish checkpoint storage on
different sides. However, I think this is more like a interface refactor which
contains more than just "avoid to send mkdirs from task sides". Shall we also
include this refactor in this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
