Savonitar commented on code in PR #28709:
URL: https://github.com/apache/flink/pull/28709#discussion_r3591070584
##########
flink-libraries/flink-state-processing-api/src/main/java/org/apache/flink/state/api/output/SavepointOutputFormat.java:
##########
@@ -79,7 +79,19 @@ public void writeRecord(CheckpointMetadata metadata) throws
IOException {
() -> {
try (CheckpointMetadataOutputStream out =
targetLocation.createMetadataOutputStream()) {
- Checkpoints.storeCheckpointMetadata(metadata,
out);
+ // Must stay on the variant WITHOUT the
exclusive directory. Files
+ // retained from an existing savepoint are
copied into this
+ // directory by FileCopyFunction, but their
handles still reference
+ // the source savepoint. Writing without the
exclusive directory
+ // keeps the relative (file-name-only)
encoding, which resolves
+ // against this directory on restore and keeps
the savepoint
+ // self-contained. The exclusive-dir-aware
+ // Checkpoints.storeCheckpointMetadata would
instead fill this
+ // savepoint's metadata with absolute paths
pointing into the
+ // source savepoint, breaking this savepoint
once the source is
+ // deleted.
+
Checkpoints.storeCheckpointMetadataWithoutExclusiveDir(
+ metadata, out);
Review Comment:
> FileCopyFunction what's that?
FileCopyFunction is a state processor API thing.
> Who is copying what? SavepointOutputFormat doesn't have to be invovled in
copy any files AFAIU.
SavepointWriter.write() builds a batch job with two sinks:
1. FileCopyFunction: copies every state file referenced by the carried-over
operator states into the new savepoint directory, keeping the file name.
2. SavepointOutputFormat: writes the _metadata. So you're right and this
class copies nothing but the other sink of the same job (FileCopyFunction)
does.
> Again I'm confused who is copying what, and why are we talking about
source in this class/method.
The "source" is the savepoint from which one we build new one, the
carried-over handles still record paths inside it, nothing rewrites them. With
`storeCheckpointMetadataWithoutExclusiveDir` the metadata stores only file
names, which resolve to the local copies on restore, so the source can be
deleted.
I updated the comment significantly with this context in
https://github.com/apache/flink/pull/28709/changes/3d0dda1a003f478641aba076e8d5feec20500eed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
