Re: connection time out

[Yuheng Du]

Also, I can see the topic "speedx2" being created in the broker, but not
message data is coming through.

On Sun, Nov 29, 2015 at 7:00 PM, Yuheng Du  wrote:

> Hi guys,
>
> I was running a single node broker in a cluster. And when I run the
> producer in another cluster, I got connection time out error.
>
> I can ping into port 9092 and other ports on the broker machine from the
> producer. I just can't publish any messages. The command I used to run the
> producer is:
>
> bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
> speedx2 50 100 -1 acks=1 bootstrap.servers=130.127.xxx.xxx:9092
> buffer.memory=67184 batch.size=8196
>
> Can anyone suggest what the problem might be?
>
>
> Thank you!
>
>
> best,
>
> Yuheng
>

connection time out

[Yuheng Du]

Hi guys,

I was running a single node broker in a cluster. And when I run the
producer in another cluster, I got connection time out error.

I can ping into port 9092 and other ports on the broker machine from the
producer. I just can't publish any messages. The command I used to run the
producer is:

bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
speedx2 50 100 -1 acks=1 bootstrap.servers=130.127.xxx.xxx:9092
buffer.memory=67184 batch.size=8196

Can anyone suggest what the problem might be?


Thank you!


best,

Yuheng

Re: Maximum Topic Length in Kafka

[Marko Bonaći]

Did not know that quotas landed in 0.9. Very nice!
Being able to throttle clients that don't have real-time SLAs (in favor of
those who do) is a great addition.
Thanks for that Grant.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sun, Nov 29, 2015 at 5:05 PM, Grant Henke  wrote:

> Quotas (KIP-13) is actually included in the recent 0.9.0 release. More
> about them can be read in the documentation here:
>
>- http://kafka.apache.org/documentation.html#design_quotas
>- http://kafka.apache.org/documentation.html#quotas
>
>
>
> On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći 
> wrote:
>
> > Yes, I thought you weren't interested in retention, but how to limit the
> > amount of messages produced into a topic.
> > Take a look at this Kafka Improvement Proposal (KIP):
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > But, AFAIK, there's nothing currently available for your use case.
> >
> > Perhaps you could check Consumer offsets from your Producer and then
> decide
> > based on that information whether to throttle Producer or not. Could get
> > complicated really fast, though.
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext  | Contact
> > 
> >
> > On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna 
> > wrote:
> >
> > > Let me explain my use case:-
> > >
> > > We have a ELK setup in which logstash-forwarders pushes logs from
> > different
> > > services to a logstash. The logstash then pushes them to kafka. The
> > > logstash consumer then pulls them out of Kafka and indexes them to
> > > Elasticsearch cluster.
> > >
> > > We are trying to ensure that no single service logs doesn't overwhelm
> the
> > > system. So I was thinking if each service logs go in their own topics
> in
> > > kafka and if we can specify a maximum length in the topic then the
> > producer
> > > of that topic can block when a kafka topic is full.
> > > AFAIK there is no such notion as maximum length of a topic, i.e. offset
> > has
> > > no limit, except Long.MAX_VALUE I think, which should be enough for a
> > > couple of lifetimes (9 * 10E18, or quintillion or million trillions).
> > >
> > > What would be the purpose of that, besides being a nice foot-gun :)
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext  | Contact
> > > 
> > >
> > > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna <
> subharaj.ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Can some one please let me know the following:-
> > > >
> > > >
> > > >1. Is it possible to specify maximum length of a particular topic
> (
> > in
> > > >terms of number of messages ) in kafka ?
> > > >2. Also how does Kafka behave when a particular topic gets full?
> > > >3. Can the producer be blocked if a topic get full rather than
> > > deleting
> > > >old messages?
> > > >
> > > > I have gone through the documentation
> > > > 
> > but
> > > > could not find anything of what I am looking for.
> > > >
> > >
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: Maximum Topic Length in Kafka

[Grant Henke]

Quotas (KIP-13) is actually included in the recent 0.9.0 release. More
about them can be read in the documentation here:

   - http://kafka.apache.org/documentation.html#design_quotas
   - http://kafka.apache.org/documentation.html#quotas



On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći 
wrote:

> Yes, I thought you weren't interested in retention, but how to limit the
> amount of messages produced into a topic.
> Take a look at this Kafka Improvement Proposal (KIP):
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> But, AFAIK, there's nothing currently available for your use case.
>
> Perhaps you could check Consumer offsets from your Producer and then decide
> based on that information whether to throttle Producer or not. Could get
> complicated really fast, though.
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext  | Contact
> 
>
> On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna 
> wrote:
>
> > Let me explain my use case:-
> >
> > We have a ELK setup in which logstash-forwarders pushes logs from
> different
> > services to a logstash. The logstash then pushes them to kafka. The
> > logstash consumer then pulls them out of Kafka and indexes them to
> > Elasticsearch cluster.
> >
> > We are trying to ensure that no single service logs doesn't overwhelm the
> > system. So I was thinking if each service logs go in their own topics in
> > kafka and if we can specify a maximum length in the topic then the
> producer
> > of that topic can block when a kafka topic is full.
> > AFAIK there is no such notion as maximum length of a topic, i.e. offset
> has
> > no limit, except Long.MAX_VALUE I think, which should be enough for a
> > couple of lifetimes (9 * 10E18, or quintillion or million trillions).
> >
> > What would be the purpose of that, besides being a nice foot-gun :)
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext  | Contact
> > 
> >
> > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
> > wrote:
> >
> > > Hi,
> > >
> > > Can some one please let me know the following:-
> > >
> > >
> > >1. Is it possible to specify maximum length of a particular topic (
> in
> > >terms of number of messages ) in kafka ?
> > >2. Also how does Kafka behave when a particular topic gets full?
> > >3. Can the producer be blocked if a topic get full rather than
> > deleting
> > >old messages?
> > >
> > > I have gone through the documentation
> > > 
> but
> > > could not find anything of what I am looking for.
> > >
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: Maximum Topic Length in Kafka

[Marko Bonaći]

Yes, I thought you weren't interested in retention, but how to limit the
amount of messages produced into a topic.
Take a look at this Kafka Improvement Proposal (KIP):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
But, AFAIK, there's nothing currently available for your use case.

Perhaps you could check Consumer offsets from your Producer and then decide
based on that information whether to throttle Producer or not. Could get
complicated really fast, though.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna 
wrote:

> Let me explain my use case:-
>
> We have a ELK setup in which logstash-forwarders pushes logs from different
> services to a logstash. The logstash then pushes them to kafka. The
> logstash consumer then pulls them out of Kafka and indexes them to
> Elasticsearch cluster.
>
> We are trying to ensure that no single service logs doesn't overwhelm the
> system. So I was thinking if each service logs go in their own topics in
> kafka and if we can specify a maximum length in the topic then the producer
> of that topic can block when a kafka topic is full.
> AFAIK there is no such notion as maximum length of a topic, i.e. offset has
> no limit, except Long.MAX_VALUE I think, which should be enough for a
> couple of lifetimes (9 * 10E18, or quintillion or million trillions).
>
> What would be the purpose of that, besides being a nice foot-gun :)
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext  | Contact
> 
>
> On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna 
> wrote:
>
> > Hi,
> >
> > Can some one please let me know the following:-
> >
> >
> >1. Is it possible to specify maximum length of a particular topic ( in
> >terms of number of messages ) in kafka ?
> >2. Also how does Kafka behave when a particular topic gets full?
> >3. Can the producer be blocked if a topic get full rather than
> deleting
> >old messages?
> >
> > I have gone through the documentation
> >  but
> > could not find anything of what I am looking for.
> >
>