Quotas (KIP-13) is actually included in the recent 0.9.0 release. More about them can be read in the documentation here:
- http://kafka.apache.org/documentation.html#design_quotas - http://kafka.apache.org/documentation.html#quotas On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći <marko.bon...@sematext.com> wrote: > Yes, I thought you weren't interested in retention, but how to limit the > amount of messages produced into a topic. > Take a look at this Kafka Improvement Proposal (KIP): > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas > But, AFAIK, there's nothing currently available for your use case. > > Perhaps you could check Consumer offsets from your Producer and then decide > based on that information whether to throttle Producer or not. Could get > complicated really fast, though. > > Marko Bonaći > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > Solr & Elasticsearch Support > Sematext <http://sematext.com/> | Contact > <http://sematext.com/about/contact.html> > > On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna <subharaj.ma...@gmail.com> > wrote: > > > Let me explain my use case:- > > > > We have a ELK setup in which logstash-forwarders pushes logs from > different > > services to a logstash. The logstash then pushes them to kafka. The > > logstash consumer then pulls them out of Kafka and indexes them to > > Elasticsearch cluster. > > > > We are trying to ensure that no single service logs doesn't overwhelm the > > system. So I was thinking if each service logs go in their own topics in > > kafka and if we can specify a maximum length in the topic then the > producer > > of that topic can block when a kafka topic is full. > > AFAIK there is no such notion as maximum length of a topic, i.e. offset > has > > no limit, except Long.MAX_VALUE I think, which should be enough for a > > couple of lifetimes (9 * 10E18, or quintillion or million trillions). > > > > What would be the purpose of that, besides being a nice foot-gun :) > > > > Marko Bonaći > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > Solr & Elasticsearch Support > > Sematext <http://sematext.com/> | Contact > > <http://sematext.com/about/contact.html> > > > > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna <subharaj.ma...@gmail.com> > > wrote: > > > > > Hi, > > > > > > Can some one please let me know the following:- > > > > > > > > > 1. Is it possible to specify maximum length of a particular topic ( > in > > > terms of number of messages ) in kafka ? > > > 2. Also how does Kafka behave when a particular topic gets full? > > > 3. Can the producer be blocked if a topic get full rather than > > deleting > > > old messages? > > > > > > I have gone through the documentation > > > <http://kafka.apache.org/081/documentation.html#basic_ops_add_topic> > but > > > could not find anything of what I am looking for. > > > > > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
