@Otis thanks for your answer.
Of course it is not about manually watching connection. We already have
monitoring tools in place. But I would expect the kafka client to provide
means for get notified about potential connectivity issues.
Actual behavior is this: the client retries to reconnect
Yes, I used TopicMetaDataRequest to determine the lead broker for the partition.
I’ve also compared the broker selected by my code with the output from
bin/kafka-topics.sh --describe[...] and as far as I can tell, it’s using the
broker which is described as leader for the topic.
/Magnus
On
Hi,
I’m not sure if I’m confusing the concepts here, because I don’t understand
your answer?
I thought that the reason for making a PartitionOffsetRequest was to determine
which offset to use when fetching messages? Also, how is message.max.bytes
related?
I’m using the simple consumer api,
Neha,
Thanks. I'd still love to know if anyone has used Consul and/or Confd to
manage a cluster.
Casey
From: Neha Narkhede [neha.narkh...@gmail.com]
Sent: Thursday, October 16, 2014 9:54 AM
To: users@kafka.apache.org
Subject: Re: Kafka/Zookeeper
Hi,
The method kafka.api.FetchRequestBuilder.maxWait(maxwait: Int) does not give
the result I would have expected. I have not been able to find any
documentation or clarification on the method’s meaning.
My assumption is: If I send a fetch request for messages after offset X for a
partition
Alex,
In the above mentioned case I think monitoring the consumer lag
will be helpful
kafka.consumer:name=([-.\w]+)-MaxLag,type=ConsumerFetcherManager.
Another approach I used before was to use metrics library to send
consumer stats to ganglia and use nagios alert on those stats, If the
Hi Balaji,
You could do a rolling bounce of the brokers to do the in-place upgrade if
your partitions have at least two replicas. After that you may probably
need to rebalance the leaders if they are not balanced any more.
Guozhang
On Wed, Oct 15, 2014 at 10:53 AM, Seshadri, Balaji
At least two including the leader?
On Fri, Oct 17, 2014 at 8:12 AM, Guozhang Wang wangg...@gmail.com wrote:
Hi Balaji,
You could do a rolling bounce of the brokers to do the in-place upgrade if
your partitions have at least two replicas. After that you may probably
need to rebalance the
Casey,
Could you describe a little more about how these would help manage a
cluster?
My understanding is that Consul provides service discovery and leader
election. Kafka already uses ZooKeeper for brokers to discover each other
and elect partition leaders. Kafka high-level consumers use ZK to
Yes.
On Fri, Oct 17, 2014 at 8:52 AM, Roger Hoover roger.hoo...@gmail.com
wrote:
At least two including the leader?
On Fri, Oct 17, 2014 at 8:12 AM, Guozhang Wang wangg...@gmail.com wrote:
Hi Balaji,
You could do a rolling bounce of the brokers to do the in-place upgrade
if
your
Is Kafka supposed to throw exception if topic doesn't exist? It appears
that there is no exception thrown even though no messages are delivered and
there are errors logged in Kafka logs.
Roger,
My understanding of both, beyond what Zookeeper already does, are:
1. Consul can be used to monitor a service and report it's status. This can be
very useful for knowing if a service, such as Zookeeper of Kafka, goes down.
This can be done through a built-in web interface.
2. Confd
Thank you, Casey. Both of those features sound pretty useful.
On Fri, Oct 17, 2014 at 10:16 AM, Sybrandy, Casey
casey.sybra...@six3systems.com wrote:
Roger,
My understanding of both, beyond what Zookeeper already does, are:
1. Consul can be used to monitor a service and report it's
If you have auto.create.topics.enable set to true (default),
producing to a topic creates it.
Its a bit tricky because the send that creates the topic can fail
with leader not found or similar issue. retrying few times will
eventually succeed as the topic gets created and the leader gets
elected.
Hi, everyone,
We are fixing the mbean names in kafka-1482, by adding separate explicit
tags in the name for things like clientId and topic. Another thing that
some people have complained before is that we use quotes in the jmx name.
Should we also just get rid of the quotes as part of kafka-1482?
0.8.1.1 producer is Sync by default, and you can set producer.type to
async if needed.
On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
Thanks! How can I tell if I am using async producer? I thought all the
sends are async in nature
On Fri, Oct 17, 2014 at 11:44 AM,
btw. I got a blog post where I show how I work around the blocking
hasNext() thing.
May be helpful:
http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
On Thu, Oct 16, 2014 at 12:52 PM, Neha Narkhede neha.narkh...@gmail.com wrote:
Josh,
The consumer's API doesn't
Little confused :) From one of the examples I am using property
request.required.acks=0,
I thought this sets the producer to be async?
On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira gshap...@cloudera.com
wrote:
0.8.1.1 producer is Sync by default, and you can set producer.type to
async if
+1 to get rid of quotes, thanks!
—
Sent from Mailbox
On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao jun...@gmail.com wrote:
Hi, everyone,
We are fixing the mbean names in kafka-1482, by adding separate explicit
tags in the name for things like clientId and topic. Another thing that
some
Its using the sync producer without waiting for any broker to acknowledge the
write. This explains the lack of errors you are seeing.
—
Sent from Mailbox
On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
Little confused :) From one of the examples I am using
Still don't understand the difference. If it's not waiting for the ack then
doesn't it make async?
On Fri, Oct 17, 2014 at 12:55 PM, gshap...@cloudera.com wrote:
Its using the sync producer without waiting for any broker to acknowledge
the write. This explains the lack of errors you are
Sorry if I'm confusing you :)
Kafka 0.8.1.1 has two producers sync and async. You are using the sync
producer without waiting for acks. I hope this helps?
Regardless, did you check if the partition got created? are you able
to produce messages? are you able to consume them?
Gwen
On Fri, Oct
+1 on getting rid of the quotes.
On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal
magnus.spang...@deltaprojects.com wrote:
+1 to get rid of quotes, thanks!
—
Sent from Mailbox
On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao jun...@gmail.com wrote:
Hi, everyone,
We are fixing the mbean
There's some good advice here: https://kafka.apache.org/081/ops.html
And you may enjoy this post too:
http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
On Fri, Oct 17, 2014 at 5:52 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
Thanks for the info. I see there are tons of
24 matches
Mail list logo
