GitHub user weiqingy added a comment to the discussion: Design proposal for #1066: dry-run / replay via savepoints
Thanks for putting this together. Savepoint-based replay seems like a promising direction, and Flink already provides a consistent snapshot of the agent’s keyed state. After looking through the current runtime, I wonder if a few recovery and isolation details need to be defined more precisely. How would the agent state map to the replay topology after replacing the live source? Flink restores state by operator ID, but the current `action-execute-operator` has no explicit UID or max parallelism. Would replay support first require stable operator identities and an explicit rule for dropping the old source state? How would a savepoint identify the exact starting point in the captured input? Flink’s exactly-once state guarantee relies on restoring both operator state and the matching source position. Since `ReplaySource` replaces the original source, would the capture path need a checkpoint-linked manifest containing the replay offsets, typed key information, serializers, and key ordering? If reproducing downstream event-time behavior is also a goal, would watermarks and source partition information need to be included as well? I also wonder whether an ordinary savepoint contains too much execution state for this use case. Flink Agents checkpoints pending `ActionTask`s, pending input events, processing keys, sequence numbers, and durable recovery markers in addition to memory. A pending task also contains the Action definition from the original job. Could a replay of changed Action logic therefore resume old logic first? Would a framework-specific quiesce-and-snapshot flow, or a derived savepoint containing only replay-safe state, provide a clearer boundary? How should the external action-state store be handled? Restoring the savepoint rebuilds action state from the configured Kafka topic or Fluss table, while new replay actions write back to that same store. Because the store keys include the business key and sequence number but no replay namespace, could running production and replay together create divergent recovery records? Would replay need a separate store with restored recovery markers removed or rewritten? Long-term memory has a similar question. The restored job identifier points to the production Mem0 namespace. A new identifier would isolate replay writes, but it would no longer expose the production long-term memory being evaluated. Would a read-only snapshot or fork of that namespace be needed? Finally, can an Action-level `idempotent/read-only` marker enforce the intended safety? Custom Actions can call HTTP services, databases, or filesystems directly, outside framework-managed resources. Would it be clearer to enforce replay policy at ChatModel, Tool, vector-store, and long-term-memory boundaries, while treating direct external calls as unsupported unless the replay job is sandboxed? Perhaps a focused first version could combine stable operator IDs, a savepoint-linked input manifest, replay-safe state projection, isolated external state, and replay-aware resource proxies. Would that give the feature a safer and more testable starting point? GitHub link: https://github.com/apache/flink-agents/discussions/1069#discussioncomment-18212224 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
