Re: Pros and cons of dockerizing kafka brokers?
Thanks, Christian. I am currently reading about kafka-on-mesos. I will hack something this weekend to see if I can bring up a kafka scheduler on mesos using dockerized brokers. . -- κρισhναν On Thu, Jul 7, 2016 at 7:29 PM, Christian Posta <christian.po...@gmail.com> wrote: > One thing I can think of is Kafka likes lots of OS page cache. Dockerizing > from the standpoint of packaging configs is a good idea, just make sure if > you're running many brokers together on the same host, they've got enough > resources (CPU/Mem) so they don't starve each other. > > On Thu, Jul 7, 2016 at 2:30 AM, Krish <krishnan.k.i...@gmail.com> wrote: > >> Hi, >> I am currently testing a custom docker volume driver plugin for AWS >> EFS/EBS >> access and mounting. So, running kafka broker inside a container makes >> will >> ease up a lot of configuration issues wrt storage for me. >> >> Are there any pros and cons of dockerizing kafka broker? >> Off the top of my head, since kafka forms the base of our setup, I can >> think of making is use the host networking stack, and increase ulimits for >> the container. >> I would like to know if and when kafka becomes greedy and cannibalizes >> resources; I can also ensure that it runs on a dedicated machine. >> >> Thanks. >> >> Best, >> Krish >> > > > > -- > *Christian Posta* > twitter: @christianposta > http://www.christianposta.com/blog > http://fabric8.io > >
Pros and cons of dockerizing kafka brokers?
Hi, I am currently testing a custom docker volume driver plugin for AWS EFS/EBS access and mounting. So, running kafka broker inside a container makes will ease up a lot of configuration issues wrt storage for me. Are there any pros and cons of dockerizing kafka broker? Off the top of my head, since kafka forms the base of our setup, I can think of making is use the host networking stack, and increase ulimits for the container. I would like to know if and when kafka becomes greedy and cannibalizes resources; I can also ensure that it runs on a dedicated machine. Thanks. Best, Krish
Re: Expired messages in kafka topic
Hi, I am trying to set up a kafka dev environment for the past few days. I followed this <https://developer.ibm.com/opentech/2016/06/06/how-to-develop-for-apache-kafka-using-eclipse/> link; the only difference being that I downloaded the latest scala IDE, instead of Eclipse. Any pointers to resolved this please. -- κρισhναν On Thu, Jun 23, 2016 at 11:06 PM, Krish <krishnan.k.i...@gmail.com> wrote: > Gwen, > Have selected priority 'minor', component as 'core', have assigned no > labels. > Jira link: https://issues.apache.org/jira/browse/KAFKA-3895. > > I have also added a question to the jira issue, alongwith a rough approach > that I have in mind. > It would be great if you can have a look and provide comments. > > I am still setting up the dev env for kafka; will update as I progress. > Thanks. > > > > -- > κρισhναν > > On Thu, Jun 23, 2016 at 9:30 PM, Gwen Shapira <g...@confluent.io> wrote: > >> Thats a pretty cool feature, if anyone feels like opening a JIRA :) >> >> On Thu, Jun 23, 2016 at 8:46 AM, Christian Posta >> <christian.po...@gmail.com> wrote: >> > Sounds like something a traditional message broker (ie, ActiveMQ) would >> be >> > able to do with a TTL setting and expiry. Expired messages get moved to >> a >> > DLQ. >> > >> > On Thu, Jun 23, 2016 at 2:45 AM, Krish <krishnan.k.i...@gmail.com> >> wrote: >> > >> >> Hi, >> >> I am trying to design a real-time application where message timeout >> can be >> >> as low as a minute or two (message can get stale real-fast). >> >> >> >> In the rare chance that the consumers lag too far behind in processing >> >> messages from the broker, is there a concept of expired message queue >> in >> >> Kafka? >> >> >> >> I would like to know if a message has expired and then park it in some >> >> topic till as such time that a service can dequeue, process it and/or >> >> investigate it. >> >> >> >> Thanks. >> >> >> >> Best, >> >> Krish >> >> >> > >> > >> > >> > -- >> > *Christian Posta* >> > twitter: @christianposta >> > http://www.christianposta.com/blog >> > http://fabric8.io >> > >
Re: Expired messages in kafka topic
Gwen, Have selected priority 'minor', component as 'core', have assigned no labels. Jira link: https://issues.apache.org/jira/browse/KAFKA-3895. I have also added a question to the jira issue, alongwith a rough approach that I have in mind. It would be great if you can have a look and provide comments. I am still setting up the dev env for kafka; will update as I progress. Thanks. -- κρισhναν On Thu, Jun 23, 2016 at 9:30 PM, Gwen Shapira <g...@confluent.io> wrote: > Thats a pretty cool feature, if anyone feels like opening a JIRA :) > > On Thu, Jun 23, 2016 at 8:46 AM, Christian Posta > <christian.po...@gmail.com> wrote: > > Sounds like something a traditional message broker (ie, ActiveMQ) would > be > > able to do with a TTL setting and expiry. Expired messages get moved to a > > DLQ. > > > > On Thu, Jun 23, 2016 at 2:45 AM, Krish <krishnan.k.i...@gmail.com> > wrote: > > > >> Hi, > >> I am trying to design a real-time application where message timeout can > be > >> as low as a minute or two (message can get stale real-fast). > >> > >> In the rare chance that the consumers lag too far behind in processing > >> messages from the broker, is there a concept of expired message queue in > >> Kafka? > >> > >> I would like to know if a message has expired and then park it in some > >> topic till as such time that a service can dequeue, process it and/or > >> investigate it. > >> > >> Thanks. > >> > >> Best, > >> Krish > >> > > > > > > > > -- > > *Christian Posta* > > twitter: @christianposta > > http://www.christianposta.com/blog > > http://fabric8.io >
Re: Expired messages in kafka topic
Tom, When you say this: "Deletion can happen at different times on the different replicas of the log, and to different messages. Whilst a consumer will only be reading from the lead broker for any log at any one time, the leader can and will change to handle broker failure." basically it means that a dead-letter Q like functionality can get multiple messages, and anyone draining the queue needs to take care of duplicates. Right so far? -- κρισhναν On Thu, Jun 23, 2016 at 9:21 PM, Krish <krishnan.k.i...@gmail.com> wrote: > Well, we are already using Kafka and would like to get this feature. > How hard can it be to hack it and use a custom kafka!? ;) > > Let me look up the source code (never have checked it) and see what can be > done. > Thanks Tom and Christian, for helping me decide fast. > > > > -- > κρισhναν > > On Thu, Jun 23, 2016 at 9:16 PM, Christian Posta < > christian.po...@gmail.com> wrote: > >> Sounds like something a traditional message broker (ie, ActiveMQ) would >> be able to do with a TTL setting and expiry. Expired messages get moved to >> a DLQ. >> >> On Thu, Jun 23, 2016 at 2:45 AM, Krish <krishnan.k.i...@gmail.com> wrote: >> >>> Hi, >>> I am trying to design a real-time application where message timeout can >>> be >>> as low as a minute or two (message can get stale real-fast). >>> >>> In the rare chance that the consumers lag too far behind in processing >>> messages from the broker, is there a concept of expired message queue in >>> Kafka? >>> >>> I would like to know if a message has expired and then park it in some >>> topic till as such time that a service can dequeue, process it and/or >>> investigate it. >>> >>> Thanks. >>> >>> Best, >>> Krish >>> >> >> >> >> -- >> *Christian Posta* >> twitter: @christianposta >> http://www.christianposta.com/blog >> http://fabric8.io >> >> >
Re: Expired messages in kafka topic
Well, we are already using Kafka and would like to get this feature. How hard can it be to hack it and use a custom kafka!? ;) Let me look up the source code (never have checked it) and see what can be done. Thanks Tom and Christian, for helping me decide fast. -- κρισhναν On Thu, Jun 23, 2016 at 9:16 PM, Christian Posta <christian.po...@gmail.com> wrote: > Sounds like something a traditional message broker (ie, ActiveMQ) would be > able to do with a TTL setting and expiry. Expired messages get moved to a > DLQ. > > On Thu, Jun 23, 2016 at 2:45 AM, Krish <krishnan.k.i...@gmail.com> wrote: > >> Hi, >> I am trying to design a real-time application where message timeout can be >> as low as a minute or two (message can get stale real-fast). >> >> In the rare chance that the consumers lag too far behind in processing >> messages from the broker, is there a concept of expired message queue in >> Kafka? >> >> I would like to know if a message has expired and then park it in some >> topic till as such time that a service can dequeue, process it and/or >> investigate it. >> >> Thanks. >> >> Best, >> Krish >> > > > > -- > *Christian Posta* > twitter: @christianposta > http://www.christianposta.com/blog > http://fabric8.io > >
Re: Expired messages in kafka topic
Thanks Tom. Is there any way a consumer can be triggered when the message is about to be deleted by Kafka? -- κρισhναν On Thu, Jun 23, 2016 at 6:16 PM, Tom Crayford <tcrayf...@heroku.com> wrote: > Hi, > > A pretty reasonable thing to do here would be to have a consumer that > moved "old" events to another topic. > > Kafka has no concept of an expired queue, the only thing it can do once a > message is aged out is delete it. The deletion is done in bulk and > typically is set to 24h or even higher (LinkedIn use 4 days, the default is > 7 days). > > Thanks > > Tom Crayford > Heroku Kafka > > On Thu, Jun 23, 2016 at 10:45 AM, Krish <krishnan.k.i...@gmail.com> wrote: > >> Hi, >> I am trying to design a real-time application where message timeout can be >> as low as a minute or two (message can get stale real-fast). >> >> In the rare chance that the consumers lag too far behind in processing >> messages from the broker, is there a concept of expired message queue in >> Kafka? >> >> I would like to know if a message has expired and then park it in some >> topic till as such time that a service can dequeue, process it and/or >> investigate it. >> >> Thanks. >> >> Best, >> Krish >> > >
Expired messages in kafka topic
Hi, I am trying to design a real-time application where message timeout can be as low as a minute or two (message can get stale real-fast). In the rare chance that the consumers lag too far behind in processing messages from the broker, is there a concept of expired message queue in Kafka? I would like to know if a message has expired and then park it in some topic till as such time that a service can dequeue, process it and/or investigate it. Thanks. Best, Krish
