Did not know that quotas landed in 0.9. Very nice! Being able to throttle clients that don't have real-time SLAs (in favor of those who do) is a great addition. Thanks for that Grant.
Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext <http://sematext.com/> | Contact <http://sematext.com/about/contact.html> On Sun, Nov 29, 2015 at 5:05 PM, Grant Henke <ghe...@cloudera.com> wrote: > Quotas (KIP-13) is actually included in the recent 0.9.0 release. More > about them can be read in the documentation here: > > - http://kafka.apache.org/documentation.html#design_quotas > - http://kafka.apache.org/documentation.html#quotas > > > > On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći <marko.bon...@sematext.com> > wrote: > > > Yes, I thought you weren't interested in retention, but how to limit the > > amount of messages produced into a topic. > > Take a look at this Kafka Improvement Proposal (KIP): > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas > > But, AFAIK, there's nothing currently available for your use case. > > > > Perhaps you could check Consumer offsets from your Producer and then > decide > > based on that information whether to throttle Producer or not. Could get > > complicated really fast, though. > > > > Marko Bonaći > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > Solr & Elasticsearch Support > > Sematext <http://sematext.com/> | Contact > > <http://sematext.com/about/contact.html> > > > > On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna <subharaj.ma...@gmail.com> > > wrote: > > > > > Let me explain my use case:- > > > > > > We have a ELK setup in which logstash-forwarders pushes logs from > > different > > > services to a logstash. The logstash then pushes them to kafka. The > > > logstash consumer then pulls them out of Kafka and indexes them to > > > Elasticsearch cluster. > > > > > > We are trying to ensure that no single service logs doesn't overwhelm > the > > > system. So I was thinking if each service logs go in their own topics > in > > > kafka and if we can specify a maximum length in the topic then the > > producer > > > of that topic can block when a kafka topic is full. > > > AFAIK there is no such notion as maximum length of a topic, i.e. offset > > has > > > no limit, except Long.MAX_VALUE I think, which should be enough for a > > > couple of lifetimes (9 * 10E18, or quintillion or million trillions). > > > > > > What would be the purpose of that, besides being a nice foot-gun :) > > > > > > Marko Bonaći > > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > > Solr & Elasticsearch Support > > > Sematext <http://sematext.com/> | Contact > > > <http://sematext.com/about/contact.html> > > > > > > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna < > subharaj.ma...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > > > > > Can some one please let me know the following:- > > > > > > > > > > > > 1. Is it possible to specify maximum length of a particular topic > ( > > in > > > > terms of number of messages ) in kafka ? > > > > 2. Also how does Kafka behave when a particular topic gets full? > > > > 3. Can the producer be blocked if a topic get full rather than > > > deleting > > > > old messages? > > > > > > > > I have gone through the documentation > > > > <http://kafka.apache.org/081/documentation.html#basic_ops_add_topic> > > but > > > > could not find anything of what I am looking for. > > > > > > > > > > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >
