Re: Expired messages in kafka topic

Hi, I am trying to set up a kafka dev environment for the past few days. I followed this link; the only difference being that I downloaded the latest scala IDE, instead of Eclipse. Any pointers to resolv

Re: Expired messages in kafka topic

Gwen, Have selected priority 'minor', component as 'core', have assigned no labels. Jira link: https://issues.apache.org/jira/browse/KAFKA-3895. I have also added a question to the jira issue, alongwith a rough approach that I have in mind. It would be great if you can have a look and provide comm

Re: Expired messages in kafka topic

Thats a pretty cool feature, if anyone feels like opening a JIRA :) On Thu, Jun 23, 2016 at 8:46 AM, Christian Posta wrote: > Sounds like something a traditional message broker (ie, ActiveMQ) would be > able to do with a TTL setting and expiry. Expired messages get moved to a > DLQ. > > On Thu, J

Re: Expired messages in kafka topic

Tom, When you say this: "Deletion can happen at different times on the different replicas of the log, and to different messages. Whilst a consumer will only be reading from the lead broker for any log at any one time, the leader can and will change to handle broker failure." basically it means tha

Re: Expired messages in kafka topic

Well, we are already using Kafka and would like to get this feature. How hard can it be to hack it and use a custom kafka!? ;) Let me look up the source code (never have checked it) and see what can be done. Thanks Tom and Christian, for helping me decide fast. -- κρισhναν On Thu, Jun 23, 2016

Re: Expired messages in kafka topic

Sounds like something a traditional message broker (ie, ActiveMQ) would be able to do with a TTL setting and expiry. Expired messages get moved to a DLQ. On Thu, Jun 23, 2016 at 2:45 AM, Krish wrote: > Hi, > I am trying to design a real-time application where message timeout can be > as low as a

Re: Expired messages in kafka topic

No, there's no control over that. The right way to do this is to keep up with the head of the topic and decide on "old" yourself in the consumer. Deletion can happen at different times on the different replicas of the log, and to different messages. Whilst a consumer will only be reading from the

Re: Expired messages in kafka topic

Thanks Tom. Is there any way a consumer can be triggered when the message is about to be deleted by Kafka? -- κρισhναν On Thu, Jun 23, 2016 at 6:16 PM, Tom Crayford wrote: > Hi, > > A pretty reasonable thing to do here would be to have a consumer that > moved "old" events to another topic. >

Re: Expired messages in kafka topic

Hi, A pretty reasonable thing to do here would be to have a consumer that moved "old" events to another topic. Kafka has no concept of an expired queue, the only thing it can do once a message is aged out is delete it. The deletion is done in bulk and typically is set to 24h or even higher (Linke

Expired messages in kafka topic

Hi, I am trying to design a real-time application where message timeout can be as low as a minute or two (message can get stale real-fast). In the rare chance that the consumers lag too far behind in processing messages from the broker, is there a concept of expired message queue in Kafka? I woul