letaoj commented on PR #138: URL: https://github.com/apache/flink-agents/pull/138#issuecomment-3310433416
Had a offline discussion with @Sxnan and here's the conclusion 1. We will use UnionState to guarantee the correctness during recovery even though for Kafka based state store, there could be read amplification happens. This should be okay for v0.1 and we will further optimized it once we launched the agent. 2. As @Sxnan mentioned, the state recovery finishes when the state is processed. Then during replay, the checkpoint could happen at any time so we should not simply clean up the entire map instead remember the last proceed sequence and then clean up upto the latest proceed sequence number -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
