[ https://issues.apache.org/jira/browse/FLINK-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Knauf updated FLINK-11133: ------------------------------------- Labels: (was: auto-closed) Priority: Not a Priority (was: Minor) > FsCheckpointStorage is unaware about S3 entropy when creating directories > ------------------------------------------------------------------------- > > Key: FLINK-11133 > URL: https://issues.apache.org/jira/browse/FLINK-11133 > Project: Flink > Issue Type: Bug > Components: FileSystems > Affects Versions: 1.7.0 > Reporter: Mark Cho > Priority: Not a Priority > > We currently use S3 for our checkpoint storage with S3 entropy enabled. > Entropy seems to be working correctly when writing out checkpoint metadata > file (entropy key is correctly stripped from `state.checkpoints.dir`) and > when writing out checkpoint data file (entropy key is correctly replaced with > random string). > However, from the logs, it seems like entropy key is not stripped or replaced > when `FsCheckpointStorage` creates directories in the following class: > [https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorage.java#L83-L85] > Should FsCheckpointStorage skip initializing mkdir calls if object store like > S3 is used since S3 doesn't have directory concept? > If we want to keep the `mkdir` calls in `FsCheckpointStorage`, we should > handle the entropy key specified in `state.checkpoints.dir`. Currently, > folder markers in S3 are being created by Hadoop FileSystem with the entropy > key in the path as a result of `mkdir` calls in `FsCheckpointStorage`). -- This message was sent by Atlassian Jira (v8.3.4#803005)