Re: Maximum Topic Length in Kafka

[Marko Bonaći]

Did not know that quotas landed in 0.9. Very nice!
Being able to throttle clients that don't have real-time SLAs (in favor of
those who do) is a great addition.
Thanks for that Grant.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sun, Nov 29, 2015 at 5:05 PM, Grant Henke  wrote:

> Quotas (KIP-13) is actually included in the recent 0.9.0 release. More
> about them can be read in the documentation here:
>
>- http://kafka.apache.org/documentation.html#design_quotas
>- http://kafka.apache.org/documentation.html#quotas
>
>
>
> On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći 
> wrote:
>
> > Yes, I thought you weren't interested in retention, but how to limit the
> > amount of messages produced into a topic.
> > Take a look at this Kafka Improvement Proposal (KIP):
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > But, AFAIK, there's nothing currently available for your use case.
> >
> > Perhaps you could check Consumer offsets from your Producer and then
> decide
> > based on that information whether to throttle Producer or not. Could get
> > complicated really fast, though.
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext  | Contact
> > 
> >
> > On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna 
> > wrote:
> >
> > > Let me explain my use case:-
> > >
> > > We have a ELK setup in which logstash-forwarders pushes logs from
> > different
> > > services to a logstash. The logstash then pushes them to kafka. The
> > > logstash consumer then pulls them out of Kafka and indexes them to
> > > Elasticsearch cluster.
> > >
> > > We are trying to ensure that no single service logs doesn't overwhelm
> the
> > > system. So I was thinking if each service logs go in their own topics
> in
> > > kafka and if we can specify a maximum length in the topic then the
> > producer
> > > of that topic can block when a kafka topic is full.
> > > AFAIK there is no such notion as maximum length of a topic, i.e. offset
> > has
> > > no limit, except Long.MAX_VALUE I think, which should be enough for a
> > > couple of lifetimes (9 * 10E18, or quintillion or million trillions).
> > >
> > > What would be the purpose of that, besides being a nice foot-gun :)
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext  | Contact
> > > 
> > >
> > > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna <
> subharaj.ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Can some one please let me know the following:-
> > > >
> > > >
> > > >1. Is it possible to specify maximum length of a particular topic
> (
> > in
> > > >terms of number of messages ) in kafka ?
> > > >2. Also how does Kafka behave when a particular topic gets full?
> > > >3. Can the producer be blocked if a topic get full rather than
> > > deleting
> > > >old messages?
> > > >
> > > > I have gone through the documentation
> > > > 
> > but
> > > > could not find anything of what I am looking for.
> > > >
> > >
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: Maximum Topic Length in Kafka

[Marko Bonaći]

Yes, I thought you weren't interested in retention, but how to limit the
amount of messages produced into a topic.
Take a look at this Kafka Improvement Proposal (KIP):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
But, AFAIK, there's nothing currently available for your use case.

Perhaps you could check Consumer offsets from your Producer and then decide
based on that information whether to throttle Producer or not. Could get
complicated really fast, though.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna 
wrote:

> Let me explain my use case:-
>
> We have a ELK setup in which logstash-forwarders pushes logs from different
> services to a logstash. The logstash then pushes them to kafka. The
> logstash consumer then pulls them out of Kafka and indexes them to
> Elasticsearch cluster.
>
> We are trying to ensure that no single service logs doesn't overwhelm the
> system. So I was thinking if each service logs go in their own topics in
> kafka and if we can specify a maximum length in the topic then the producer
> of that topic can block when a kafka topic is full.
> AFAIK there is no such notion as maximum length of a topic, i.e. offset has
> no limit, except Long.MAX_VALUE I think, which should be enough for a
> couple of lifetimes (9 * 10E18, or quintillion or million trillions).
>
> What would be the purpose of that, besides being a nice foot-gun :)
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext  | Contact
> 
>
> On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
> wrote:
>
> > Hi,
> >
> > Can some one please let me know the following:-
> >
> >
> >1. Is it possible to specify maximum length of a particular topic ( in
> >terms of number of messages ) in kafka ?
> >2. Also how does Kafka behave when a particular topic gets full?
> >3. Can the producer be blocked if a topic get full rather than
> deleting
> >old messages?
> >
> > I have gone through the documentation
> >  but
> > could not find anything of what I am looking for.
> >
>

Re: Maximum Topic Length in Kafka

[Grant Henke]

Quotas (KIP-13) is actually included in the recent 0.9.0 release. More
about them can be read in the documentation here:

   - http://kafka.apache.org/documentation.html#design_quotas
   - http://kafka.apache.org/documentation.html#quotas



On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći 
wrote:

> Yes, I thought you weren't interested in retention, but how to limit the
> amount of messages produced into a topic.
> Take a look at this Kafka Improvement Proposal (KIP):
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> But, AFAIK, there's nothing currently available for your use case.
>
> Perhaps you could check Consumer offsets from your Producer and then decide
> based on that information whether to throttle Producer or not. Could get
> complicated really fast, though.
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext  | Contact
> 
>
> On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna 
> wrote:
>
> > Let me explain my use case:-
> >
> > We have a ELK setup in which logstash-forwarders pushes logs from
> different
> > services to a logstash. The logstash then pushes them to kafka. The
> > logstash consumer then pulls them out of Kafka and indexes them to
> > Elasticsearch cluster.
> >
> > We are trying to ensure that no single service logs doesn't overwhelm the
> > system. So I was thinking if each service logs go in their own topics in
> > kafka and if we can specify a maximum length in the topic then the
> producer
> > of that topic can block when a kafka topic is full.
> > AFAIK there is no such notion as maximum length of a topic, i.e. offset
> has
> > no limit, except Long.MAX_VALUE I think, which should be enough for a
> > couple of lifetimes (9 * 10E18, or quintillion or million trillions).
> >
> > What would be the purpose of that, besides being a nice foot-gun :)
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext  | Contact
> > 
> >
> > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
> > wrote:
> >
> > > Hi,
> > >
> > > Can some one please let me know the following:-
> > >
> > >
> > >1. Is it possible to specify maximum length of a particular topic (
> in
> > >terms of number of messages ) in kafka ?
> > >2. Also how does Kafka behave when a particular topic gets full?
> > >3. Can the producer be blocked if a topic get full rather than
> > deleting
> > >old messages?
> > >
> > > I have gone through the documentation
> > > 
> but
> > > could not find anything of what I am looking for.
> > >
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Maximum Topic Length in Kafka

[Debraj Manna]

Hi,

Can some one please let me know the following:-


   1. Is it possible to specify maximum length of a particular topic ( in
   terms of number of messages ) in kafka ?
   2. Also how does Kafka behave when a particular topic gets full?
   3. Can the producer be blocked if a topic get full rather than deleting
   old messages?

I have gone through the documentation
 but
could not find anything of what I am looking for.

Re: Maximum Topic Length in Kafka

[Marko Bonaći]

AFAIK there is no such notion as maximum length of a topic, i.e. offset has
no limit, except Long.MAX_VALUE I think, which should be enough for a
couple of lifetimes (9 * 10E18, or quintillion or million trillions).

What would be the purpose of that, besides being a nice foot-gun :)

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
wrote:

> Hi,
>
> Can some one please let me know the following:-
>
>
>1. Is it possible to specify maximum length of a particular topic ( in
>terms of number of messages ) in kafka ?
>2. Also how does Kafka behave when a particular topic gets full?
>3. Can the producer be blocked if a topic get full rather than deleting
>old messages?
>
> I have gone through the documentation
>  but
> could not find anything of what I am looking for.
>

Re: Maximum Topic Length in Kafka

[Guozhang Wang]

Kafka server has a data retention policy based on either time or #.message
(e.g. Kafka brokers will automatically delete the oldest data segment if
its oldest data has been xx milliseconds ago, of if its total log size has
exceed yy MBs, with threshold values configurable).

The producer clients will not get affected by any of the log retention
policies on brokers, it will always be able to produce as long as the
server is alive. It is the server's responsibility to truncate data
according to the retention policy.

Guozhang

On Sat, Nov 28, 2015 at 10:34 AM, Marko Bonaći 
wrote:

> AFAIK there is no such notion as maximum length of a topic, i.e. offset has
> no limit, except Long.MAX_VALUE I think, which should be enough for a
> couple of lifetimes (9 * 10E18, or quintillion or million trillions).
>
> What would be the purpose of that, besides being a nice foot-gun :)
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext  | Contact
> 
>
> On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
> wrote:
>
> > Hi,
> >
> > Can some one please let me know the following:-
> >
> >
> >1. Is it possible to specify maximum length of a particular topic ( in
> >terms of number of messages ) in kafka ?
> >2. Also how does Kafka behave when a particular topic gets full?
> >3. Can the producer be blocked if a topic get full rather than
> deleting
> >old messages?
> >
> > I have gone through the documentation
> >  but
> > could not find anything of what I am looking for.
> >
>



-- 
-- Guozhang

Re: Maximum Topic Length in Kafka

[Debraj Manna]

Let me explain my use case:-

We have a ELK setup in which logstash-forwarders pushes logs from different
services to a logstash. The logstash then pushes them to kafka. The
logstash consumer then pulls them out of Kafka and indexes them to
Elasticsearch cluster.

We are trying to ensure that no single service logs doesn't overwhelm the
system. So I was thinking if each service logs go in their own topics in
kafka and if we can specify a maximum length in the topic then the producer
of that topic can block when a kafka topic is full.
AFAIK there is no such notion as maximum length of a topic, i.e. offset has
no limit, except Long.MAX_VALUE I think, which should be enough for a
couple of lifetimes (9 * 10E18, or quintillion or million trillions).

What would be the purpose of that, besides being a nice foot-gun :)

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
wrote:

> Hi,
>
> Can some one please let me know the following:-
>
>
>1. Is it possible to specify maximum length of a particular topic ( in
>terms of number of messages ) in kafka ?
>2. Also how does Kafka behave when a particular topic gets full?
>3. Can the producer be blocked if a topic get full rather than deleting
>old messages?
>
> I have gone through the documentation
>  but
> could not find anything of what I am looking for.
>