It does have a connection with RIP-30, compact topic can speed up state recovery.
I add some explanation to illustrate how compact topic work on state recovery in the doc. dongeforever <[email protected]> 于2022年2月10日周四 17:59写道： > What is the relationship between this RIP and RIP-30 ? > > If this RIP is based on RIP-30, it is better to attach some description. > > If this RIP is independent, it needs to explain the detailed relationship. > > > ze Ni <[email protected]> 于2022年2月10日周四 14:42写道： > > > Thanks for your review, and I am glad to add you into shepherds list. > > > > > > Exactly-once involves a larger range, such as when we should > > submit consumption offset and store state data, and what it's behavior is > > when RocketMQ-streams recover from crash. This RIP only discusses a part > of > > them, to achieve exactly-once how we should store and recover state with > > RocketMQ. > > > > When a message is found from a new shard, RocketMQ-streams load it's > > corresponding state.When shard removed because of rebalancing, > > RocketMQ-streams save state to RocketMQ and remove local. When making a > > checkpoint, firstly save state to RocketMQ, secondly commit state > > consumption offset, and at last, poll new state into local. > > > > When RocketMQ-streams crash between checkpoints, state data in RocketMQ > > exactly correspond to the last checkpoint, and state consumption offset > > corresponds to the penultimate checkpoint. > > > > > > That is to say, we can recover state from the last checkpoint when > > RocketMQ-streams restart, which is state behavior in exactly-once > > semantics. > > > > As for the complete exact-once implementation, I would like to discuss > with > > another RIP. > > > > vongosling <[email protected]> 于2022年2月10日周四 08:55写道： > > > > > I like IStorage design, good to start. This is a good improvement for > our > > > streams architecture. If you need another shepherd, you could count me > > :-) > > > > > > BTW, your design seems to not solve the exactly-once semantic in the > > > streaming scenario, or say, we miss some detailed comments in the doc? > > > > > > 倪泽 <[email protected]> 于2022年2月9日周三 17:38写道： > > > > > > > Hi, RocketMQ Community, > > > > > > > > First of all, Happy Chinese New Year! > > > > I want to start a RIP to replace mysql with RocketMQ as state store > in > > > > RocketMQ-streams. Base the RocketMQ storage, I will provide a > > > exactly-once > > > > supported and efficient state storage scheme. > > > > > > > > I have written my proposal and you can click on the link below： > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing > > > > > > > > Chinese version： > > > > > > > > https://shimo.im/docs/pjDRKHqtdQC9RWCg > > > > > > > > If you have any questions or suggestions, please reply to this email > or > > > > comment on the proposal. > > > > > > > > > > > > Thanks > > > > Nize > > > > > > > > > > > > > -- > > > Best Regards :-) > > > > > >
