GitHub user raboof added a comment to the discussion: Pekko Connector Fault Tolerance Processing
As discussed in https://github.com/apache/pekko-connectors/discussions/814 before, you'll have to decide for yourself how you want to achieve exactly-once processing for your system. Indeed you'd have to keep state somewhere - either using Pekko Persistence (e.g. https://pekko.apache.org/docs/pekko/current/typed/index-persistence-durable-state.html). But as you're already working with HDFS and S3, perhaps you could store something or derive where to continue from there? GitHub link: https://github.com/apache/pekko-connectors/discussions/834#discussioncomment-10775791 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
