JingsongLi commented on code in PR #9535:
URL: https://github.com/apache/paimon/pull/9535#discussion_r4291160156
##########
paimon-core/src/main/java/org/apache/paimon/table/source/DataTableStreamScan.java:
##########
@@ -315,6 +317,45 @@ public Long watermark() {
@Override
public void restore(@Nullable Long nextSnapshotId) {
+ if (nextSnapshotId != null) {
+ Long earliestSnapshotId = snapshotManager.earliestSnapshotId();
+ if (earliestSnapshotId != null && earliestSnapshotId >
nextSnapshotId) {
+ // The restored snapshot has already been expired. Whether the
consumer can still
+ // resume from it depends on the changelog lifecycle.
+ if (options.changelogLifecycleDecoupled()
+ &&
changelogManager.longLivedChangelogExists(nextSnapshotId)) {
Review Comment:
[P1] Apply the compactor fallback before preserving long-lived changelog
This early return also applies to `COMPACT_BUCKET_TABLE`, whose follow-up
scanner is always `DeltaFollowUpScanner`. Retaining a long-lived changelog does
not guarantee that the data delta manifests needed by that scanner still exist:
with `changelog-producer=input`, snapshot expiration deletes those manifests
while retaining the changelog metadata and changelog manifests.
I reproduced this on the current head with snapshot retention min/max set to
`2` and `changelog.num-retained.max=10`. After five APPEND commits, snapshots
4/5 remain and changelog 2 exists. Restoring the dedicated compactor at
snapshot 2 enters this branch, then `plan()` fails with `FileNotFoundException`
while reading the deleted delta manifest. The job therefore still cannot
recover in this valid configuration.
Both controls pass: an ordinary consumer can read changelog 2, and resetting
the compactor cursor to `null` lets its starting scanner advance to retained
snapshot 4. Please prioritize dedicated-compactor recovery over this
changelog-preservation branch, or restrict preservation to scans whose required
manifests are retained. Add a regression that calls `plan()` after restore; the
current checkpoint-only assertion does not cover this failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
