Yes, I thought you weren't interested in retention, but how to limit the amount of messages produced into a topic. Take a look at this Kafka Improvement Proposal (KIP): https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas But, AFAIK, there's nothing currently available for your use case.
Perhaps you could check Consumer offsets from your Producer and then decide based on that information whether to throttle Producer or not. Could get complicated really fast, though. Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext <http://sematext.com/> | Contact <http://sematext.com/about/contact.html> On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna <subharaj.ma...@gmail.com> wrote: > Let me explain my use case:- > > We have a ELK setup in which logstash-forwarders pushes logs from different > services to a logstash. The logstash then pushes them to kafka. The > logstash consumer then pulls them out of Kafka and indexes them to > Elasticsearch cluster. > > We are trying to ensure that no single service logs doesn't overwhelm the > system. So I was thinking if each service logs go in their own topics in > kafka and if we can specify a maximum length in the topic then the producer > of that topic can block when a kafka topic is full. > AFAIK there is no such notion as maximum length of a topic, i.e. offset has > no limit, except Long.MAX_VALUE I think, which should be enough for a > couple of lifetimes (9 * 10E18, or quintillion or million trillions). > > What would be the purpose of that, besides being a nice foot-gun :) > > Marko Bonaći > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > Solr & Elasticsearch Support > Sematext <http://sematext.com/> | Contact > <http://sematext.com/about/contact.html> > > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna <subharaj.ma...@gmail.com> > wrote: > > > Hi, > > > > Can some one please let me know the following:- > > > > > > 1. Is it possible to specify maximum length of a particular topic ( in > > terms of number of messages ) in kafka ? > > 2. Also how does Kafka behave when a particular topic gets full? > > 3. Can the producer be blocked if a topic get full rather than > deleting > > old messages? > > > > I have gone through the documentation > > <http://kafka.apache.org/081/documentation.html#basic_ops_add_topic> but > > could not find anything of what I am looking for. > > >
