Also, you can set a retention period and have messages get auto deleted after a certain time (default 1 week)
On Sat, Oct 8, 2016 at 3:21 AM, Hans Jespersen <[email protected]> wrote: > Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging > rethought as a distributed commit log”. The messages in the log do not get > deleted just because “all" clients have consumed the messages. Besides you > could always have a late joining consumer come along and if you mistakenly > deleted the messages then they wouldn’t be there for any new consumers. > Ditto for replaying messages back to consumers after they consume the > messages the first time. Kafka just keeps the messages and once consumed > each client moves their offset forward to point to the next message. You > can tell is any consumer has finished processing a message by checking the > offsets for that client. For the older clients these are stored in > Zookeeper but for the new consumers (0.9+) they are in a special kafka > topic dedicated to storing client offsets. > > > -hans > > > > > On Oct 7, 2016, at 1:34 PM, Hysi, Lorenc <[email protected]> wrote: > > > > Hello, > > > > Thank you for your time. > > > > I wanted to ask whether it's possible to remove a message from a topic > after making sure all consumers have gotten a it. If so what is the best > way to achieve this? > > > > Also how do I make sure that all consumers have received a message. Any > way to do this in kafka or zookeeper? I keep looking but I'm only seeing > that only consumers themselves keep track of where they are and if so any > way to have consumers talk to each other to make sure they all have > consumed the same message. > > > > If this is not the right place to ask this question I apologize in > advance and please let me know where to ask if you can. > > > > Thank you! > > > > > >
