prateekm commented on code in PR #1676:
URL: https://github.com/apache/samza/pull/1676#discussion_r1287652768
##########
samza-core/src/main/java/org/apache/samza/storage/blobstore/BlobStoreRestoreManager.java:
##########
@@ -119,13 +119,18 @@ public BlobStoreRestoreManager(TaskModel taskModel,
ExecutorService restoreExecu
@Override
public void init(Checkpoint checkpoint) {
+ init(checkpoint, false);
+ }
+
+ public void init(Checkpoint checkpoint, Boolean getDeletedBlob) {
long startTime = System.nanoTime();
LOG.debug("Initializing blob store restore manager for task: {}",
taskName);
blobStoreManager.init();
// get previous SCMs from checkpoint
- prevStoreSnapshotIndexes = blobStoreUtil.getStoreSnapshotIndexes(jobName,
jobId, taskName, checkpoint, storesToRestore);
+ prevStoreSnapshotIndexes =
+ blobStoreUtil.getStoreSnapshotIndexes(jobName, jobId, taskName,
checkpoint, storesToRestore, getDeletedBlob);
Review Comment:
Discussed offline. We use "snapshot index is deleted" as a indicator for
"files are deleted" (since that's the last step in the post-commit cleanup).
Making this check upfront allows us to keep the shouldRestore optimization
below for the happy path. If snapshot index is deleted, then all files need to
be restored with getDeletedBlob = true and restored, regardless of whether
there is a local checkpoint or not. Resolving this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
