Re: Deleting a message after all consumer have consumed it
Also, you can set a retention period and have messages get auto deleted after a certain time (default 1 week) On Sat, Oct 8, 2016 at 3:21 AM, Hans Jespersen wrote: > Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging > rethought as a distributed commit log”. The messages in the log do not get > deleted just because “all" clients have consumed the messages. Besides you > could always have a late joining consumer come along and if you mistakenly > deleted the messages then they wouldn’t be there for any new consumers. > Ditto for replaying messages back to consumers after they consume the > messages the first time. Kafka just keeps the messages and once consumed > each client moves their offset forward to point to the next message. You > can tell is any consumer has finished processing a message by checking the > offsets for that client. For the older clients these are stored in > Zookeeper but for the new consumers (0.9+) they are in a special kafka > topic dedicated to storing client offsets. > > > -hans > > > > > On Oct 7, 2016, at 1:34 PM, Hysi, Lorenc wrote: > > > > Hello, > > > > Thank you for your time. > > > > I wanted to ask whether it's possible to remove a message from a topic > after making sure all consumers have gotten a it. If so what is the best > way to achieve this? > > > > Also how do I make sure that all consumers have received a message. Any > way to do this in kafka or zookeeper? I keep looking but I'm only seeing > that only consumers themselves keep track of where they are and if so any > way to have consumers talk to each other to make sure they all have > consumed the same message. > > > > If this is not the right place to ask this question I apologize in > advance and please let me know where to ask if you can. > > > > Thank you! > > > > > >
Re: Deleting a message after all consumer have consumed it
Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging rethought as a distributed commit log”. The messages in the log do not get deleted just because “all" clients have consumed the messages. Besides you could always have a late joining consumer come along and if you mistakenly deleted the messages then they wouldn’t be there for any new consumers. Ditto for replaying messages back to consumers after they consume the messages the first time. Kafka just keeps the messages and once consumed each client moves their offset forward to point to the next message. You can tell is any consumer has finished processing a message by checking the offsets for that client. For the older clients these are stored in Zookeeper but for the new consumers (0.9+) they are in a special kafka topic dedicated to storing client offsets. -hans > On Oct 7, 2016, at 1:34 PM, Hysi, Lorenc wrote: > > Hello, > > Thank you for your time. > > I wanted to ask whether it's possible to remove a message from a topic after > making sure all consumers have gotten a it. If so what is the best way to > achieve this? > > Also how do I make sure that all consumers have received a message. Any way > to do this in kafka or zookeeper? I keep looking but I'm only seeing that > only consumers themselves keep track of where they are and if so any way to > have consumers talk to each other to make sure they all have consumed the > same message. > > If this is not the right place to ask this question I apologize in advance > and please let me know where to ask if you can. > > Thank you! > >
Deleting a message after all consumer have consumed it
Hello, Thank you for your time. I wanted to ask whether it's possible to remove a message from a topic after making sure all consumers have gotten a it. If so what is the best way to achieve this? Also how do I make sure that all consumers have received a message. Any way to do this in kafka or zookeeper? I keep looking but I'm only seeing that only consumers themselves keep track of where they are and if so any way to have consumers talk to each other to make sure they all have consumed the same message. If this is not the right place to ask this question I apologize in advance and please let me know where to ask if you can. Thank you!
