Mark Cho created FLINK-11196:
--------------------------------

             Summary: Extend S3 EntropyInjector to use key replacement (instead 
of key removal) when creating checkpoint metadata files
                 Key: FLINK-11196
                 URL: https://issues.apache.org/jira/browse/FLINK-11196
             Project: Flink
          Issue Type: Improvement
          Components: FileSystem
    Affects Versions: 1.7.0
            Reporter: Mark Cho


We currently use S3 entropy injection when writing out checkpoint data.

We also use external checkpoints so that we can resume from a checkpoint 
metadata file later.

The current implementation of S3 entropy injector makes it difficult to locate 
the checkpoint metadata files since in the newer versions of Flink, 
`state.checkpoints.dir` configuration controls where the metadata and state 
files are written, instead of having two separate paths (one for metadata, one 
for state files).

With entropy injection, we replace the entropy marker in the path specified by 
`state.checkpoints.dir` with entropy (for state files) or we strip out the 
marker (for metadata files).

 

We need to extend the entropy injection so that we can replace the entropy 
marker with a predictable path (instead of removing it) so that we can do a 
prefix query for just the metadata files.

By not using the entropy key replacement (defaults to empty string), you get 
the same behavior as it is today (entropy marker removed).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to