As part of implementing the kafka subscriber interface, I am wondering is there anyone use DL stream for storing the offsets?
For example, if I have N streams (0..N-1), I need to track the read offset for each stream and store them somewhere. I can probably use other external services (like any key/value store) to store the offset. But it would introduce extra dependencies. I am thinking if I have a map of <stream, offset> and periodically flush the map into a separate stream (let's say __offset_ stream). With proper truncation/checkpoint mechanism, it would be very fast. This use case here is a very standard replicated state machine. I am also wondering do you guys think of providing some common library on distributedlog to simply implementing state machines. - KN
