[
https://issues.apache.org/jira/browse/FLINK-40081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104728#comment-18104728
]
Rui Fan edited comment on FLINK-40081 at 8/14/26 8:08 AM:
----------------------------------------------------------
Merged to master(2.4.0) via: 516a8f95b9f59a1f56c5c6ae1a6a4ad1d8cb1e17 and
2f3c205e9266cb30240eb7f4fdab15cad629a70f
was (Author: fanrui):
Merged to master(2.4.0) via: 516a8f95b9f59a1f56c5c6ae1a6a4ad1d8cb1e17
> Enable the spilling backend for checkpointing during recovery
> -------------------------------------------------------------
>
> Key: FLINK-40081
> URL: https://issues.apache.org/jira/browse/FLINK-40081
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
> Reporter: Rui Fan
> Assignee: Rui Fan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
>
> Swap the memory backend for the disk backend behind
> execution.checkpointing.during-recovery.enabled: recovered channel state is
> fetched (and rescale-filtered) into spill files, the drainer pushes it into
> the recovery-state channels while serving as the recovery-checkpoint trigger,
> and checkpoints taken during recovery snapshot the not-yet-drained slice.
> Scope:
> * Handler factory returns the spilling handlers when the flag is on; the v1
> in-memory filtering path is deleted.
> * StreamTask flag-on recovery becomes fetch → requestPartitions →
> build/install drainer → drain → NO_OP; the transitional in-memory trigger and
> conversion-time push are deleted.
> * The unbounded heap fallback in RecoveredInputChannel#requestBufferBlocking
> is removed — spilling supersedes it, eliminating the OOM risk of the memory
> backend.
> * RecordFilterContext hardened for minimal and batch environments (spill tmp
> directories enforced only when the flag is on).
> * IT cases: unaligned checkpoint during recovery, rescale-filter
> large-record OOM regression, non-source recovery race smoke test.
> The randomized ITCase fleet (which runs the flag in ~50% of runs) must stay
> green across the backend swap — the acceptance bar for spilling being a
> drop-in replacement of the memory backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
