Re: Produce 1 million events/seconds
btw -- curious to know how well your kafka broker handles the load...
please do inform us your result.
Regards,
Nitin Kumar Sharma.
On Mon, Dec 22, 2014 at 9:52 AM, nitin sharma kumarsharma.ni...@gmail.com
wrote:
Hey Pramod,
few things:
a. You can keep 2 brokers but you can increase the ProducerSend thread on
your producer side to push more messages. best way try to create more
threads that execute the loop where send is called.
b. try to avoid/reduce putting any logic computation in the while loop ..
You can try using for loop instead of while :.
for(int i=0; imessageCount;i++)
{
KeyedMessageString, String message =
new KeyedMessageString, String(topic, i, msg);
producer.send(message);
}
Regards,
Nitin Kumar Sharma.
On Sun, Dec 21, 2014 at 11:18 PM, Pramod Deshmukh dpram...@gmail.com
wrote:
*Kafka: *Apache Kafka 0.8.1.1
*SImplePartitioner.java*
public int partition(Object key, int a_numPartitions) {
int partition = Integer.parseInt((String)key);
LOG.debug(SimplePartitioner Partion: + partition);
return partition;
}
On Sun, Dec 21, 2014 at 10:54 PM, Pramod Deshmukh dpram...@gmail.com
wrote:
I have a requirement to prove kafka producer can produce 1 million
events/second to Kafka cluster.
So far, best I could achieve is 200k events/sec on topic with 2
partitions. The latency increases with adding more partitions so I want
to
test with 2 partitions for now.
Below are the details along with produce code (java). How can I achieve
produce 1million event/sec.? I went thru kafka benchmarking blog as
well.
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
*Kafka cluster:* 3 brokers on 3 servers. Each sever is 16 TB (16 JBODs),
64GB RAM.
*Broker:* Allocated 6GB, 16 io.threads, 8 network threads.
*Topic: 2* partition, replication factor of 1 (Get high latency)
*Zookeepers: *3 zk instances running individually on master nodes (not
co-located with kafka broker/servers)
*Producer Code:*
public class TestProducer {
private static String msg = TEST KAFKA PERFORMANCE;
private static Logger LOG = Logger.getLogger(TestProducer.class);
public static void main(String... args){
System.out.println(START - Test Producer);
long messageCount = Long.parseLong(args[0]);
long messageCountForStat = Long.parseLong(args[0]);
String topic = args[1];
String brokerList = args[2];
int batchCount = Integer.parseInt(args[3]);
int topicPartions = Integer.parseInt(args[4]);
ProducerString, String producer = getProducer(brokerList,
batchCount);
Date startTime = new Date(System.currentTimeMillis());
Random rnd = new Random();
String partition = ;
//Produce messages.
while (messageCount != 0) {
partition = +(int)messageCount%topicPartions;
KeyedMessageString, String message =
new KeyedMessageString, String(topic, partition,
msg);
producer.send(message);
messageCount--;
}
Date endTime = new Date(System.currentTimeMillis());
System.out.println(#);
System.out.println(MESSAGES SENT: + messageCountForStat);
System.out.println(START TIME: + startTime);
System.out.println(END TIME: + endTime);
System.out.println(#);
System.out.println(END - Test Producer);
}
public static ProducerString, String getProducer(String
brokerList,
int batchSize) {
props.put(metadata.broker.list, brokerList);
props.put(serializer.class, kafka.serializer.StringEncoder);
props.put(partitioner.class, com.my.SimplePartitioner);
props.put(request.required.acks, 0);
props.put(producer.type, async);
props.put(compression.codec, snappy);
props.put(batch.num.messages, Integer.toString(batchSize));
ProducerConfig config = new ProducerConfig(props);
ProducerString, String producer = new ProducerString,
String(config);
return producer;
}
}
Re: Produce 1 million events/seconds
Hey Pramod,
few things:
a. You can keep 2 brokers but you can increase the ProducerSend thread on
your producer side to push more messages. best way try to create more
threads that execute the loop where send is called.
b. try to avoid/reduce putting any logic computation in the while loop ..
You can try using for loop instead of while :.
for(int i=0; imessageCount;i++)
{
KeyedMessageString, String message =
new KeyedMessageString, String(topic, i, msg);
producer.send(message);
}
Regards,
Nitin Kumar Sharma.
On Sun, Dec 21, 2014 at 11:18 PM, Pramod Deshmukh dpram...@gmail.com
wrote:
*Kafka: *Apache Kafka 0.8.1.1
*SImplePartitioner.java*
public int partition(Object key, int a_numPartitions) {
int partition = Integer.parseInt((String)key);
LOG.debug(SimplePartitioner Partion: + partition);
return partition;
}
On Sun, Dec 21, 2014 at 10:54 PM, Pramod Deshmukh dpram...@gmail.com
wrote:
I have a requirement to prove kafka producer can produce 1 million
events/second to Kafka cluster.
So far, best I could achieve is 200k events/sec on topic with 2
partitions. The latency increases with adding more partitions so I want
to
test with 2 partitions for now.
Below are the details along with produce code (java). How can I achieve
produce 1million event/sec.? I went thru kafka benchmarking blog as well.
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
*Kafka cluster:* 3 brokers on 3 servers. Each sever is 16 TB (16 JBODs),
64GB RAM.
*Broker:* Allocated 6GB, 16 io.threads, 8 network threads.
*Topic: 2* partition, replication factor of 1 (Get high latency)
*Zookeepers: *3 zk instances running individually on master nodes (not
co-located with kafka broker/servers)
*Producer Code:*
public class TestProducer {
private static String msg = TEST KAFKA PERFORMANCE;
private static Logger LOG = Logger.getLogger(TestProducer.class);
public static void main(String... args){
System.out.println(START - Test Producer);
long messageCount = Long.parseLong(args[0]);
long messageCountForStat = Long.parseLong(args[0]);
String topic = args[1];
String brokerList = args[2];
int batchCount = Integer.parseInt(args[3]);
int topicPartions = Integer.parseInt(args[4]);
ProducerString, String producer = getProducer(brokerList,
batchCount);
Date startTime = new Date(System.currentTimeMillis());
Random rnd = new Random();
String partition = ;
//Produce messages.
while (messageCount != 0) {
partition = +(int)messageCount%topicPartions;
KeyedMessageString, String message =
new KeyedMessageString, String(topic, partition,
msg);
producer.send(message);
messageCount--;
}
Date endTime = new Date(System.currentTimeMillis());
System.out.println(#);
System.out.println(MESSAGES SENT: + messageCountForStat);
System.out.println(START TIME: + startTime);
System.out.println(END TIME: + endTime);
System.out.println(#);
System.out.println(END - Test Producer);
}
public static ProducerString, String getProducer(String brokerList,
int batchSize) {
props.put(metadata.broker.list, brokerList);
props.put(serializer.class, kafka.serializer.StringEncoder);
props.put(partitioner.class, com.my.SimplePartitioner);
props.put(request.required.acks, 0);
props.put(producer.type, async);
props.put(compression.codec, snappy);
props.put(batch.num.messages, Integer.toString(batchSize));
ProducerConfig config = new ProducerConfig(props);
ProducerString, String producer = new ProducerString,
String(config);
return producer;
}
}
Kafka in C#
We are using kafka version 0.8.1.1 and trying to produce from a C# app. I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available. I am looking for a solution that is a) stable enough for production environment and b) performs well. 1) A couple of active github projects are available along with a few forks each. Has Anybody worked on either of the following two options to contribute their findings? https://github.com/Jroland/kafka-net https://github.com/miknil/Kafka4n 2) Also there is the option of IKVM to import kafka JARS into a .net DLL. Maybe even another solution? Thanks MATTI
Re: I have a problem about kafka (0.7.2)
Hmm, in 0.7, each producer is supposed to have only one ZK connection. Do you see ZK session expiration in the producer log? Also, 0.7 is pretty old and will no longer be patched. So, you probably want to try the latest 0.8.1.1. Thanks, Jun On Tue, Dec 16, 2014 at 11:53 PM, Zhao GuoHao(媒体产品技术中心) guohaozhao116...@sohu-inc.com wrote: Hello kafka : I find a problem about kafka , I build a kafka cluster with 3 zookeepers and 7 kafka brokers, it works well when developing ,but when I deploy it online ,I find that :each kafka producer keeps a lot of connections with zookeeper . On a kafka producer machine ,I use ` ss –anop | grep 2181 | wc –l ` , it has more 2000 connections .most of them like this : ESTAB 0 0:::192.168.109.144:40378 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40376 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40380 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40352 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40359 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40356 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40360 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40367 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40338 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40347 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40346 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40345 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40344 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40348 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40321 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40407 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40404 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40411 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40410 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40408 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40415 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40414 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40413 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40387 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40384 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40389 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40388 ::: 10.16.0.172:2181 ESTAB 0 0:::192.168.109.144:40396 ::: 10.16.0.172:2181 Please help me !! Thank you ! ——— 媒体技术中心 赵国昊 8187
Re: kafka-producer-perf-test.sh
What kind of load testing do you plan to do? Thanks, Jun On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi Team, I have to make a decision on whether i should go with Kafka producer test utility or build my own java tool for my load testing . Kindly let me know if anyone knows any limitation with kafka-producer-per-test.sh when it come to simulation of messages under load condition? Regards, Nitin Kumar Sharma.
Re: Kafka in C#
Another option is a HTTP wrapper around the actual producer and doing a HTTP POST from C# to a REST server e.g. https://github.com/stealthly/dropwizard-kafka-http which I know folks have done successfully. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Dec 22, 2014 at 10:55 AM, Matti Waarna mwaa...@sapient.com wrote: We are using kafka version 0.8.1.1 and trying to produce from a C# app. I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available. I am looking for a solution that is a) stable enough for production environment and b) performs well. 1) A couple of active github projects are available along with a few forks each. Has Anybody worked on either of the following two options to contribute their findings? https://github.com/Jroland/kafka-net https://github.com/miknil/Kafka4n 2) Also there is the option of IKVM to import kafka JARS into a .net DLL. Maybe even another solution? Thanks MATTI
Re: kafka-producer-perf-test.sh
have to conduce load and stress test by publishing at least 10K messages/sec to my Kafka brokers. Here is the setup: Number of brokers : 2 Topic : 1 Partition :2. Each broker leader of one. I would like to know if i can use kafka-producer-per-test.sh in first place. also, if any one knows the limitation of this utility, so that i can think of writing my own in order to overcome the limitations. Regards, Nitin Kumar Sharma. On Mon, Dec 22, 2014 at 10:57 AM, Jun Rao j...@confluent.io wrote: What kind of load testing do you plan to do? Thanks, Jun On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi Team, I have to make a decision on whether i should go with Kafka producer test utility or build my own java tool for my load testing . Kindly let me know if anyone knows any limitation with kafka-producer-per-test.sh when it come to simulation of messages under load condition? Regards, Nitin Kumar Sharma.
Re: ERROR [ReplicaFetcherThread-0-202], Error for partition
You may want to take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-WhydoIseeerror Shouldnotsetlogendoffsetonpartitioninthebrokerlog? Thanks, Jun On Wed, Dec 17, 2014 at 9:50 PM, Terry Cumaranatunge cumar...@gmail.com wrote: Hello, I'm using 0.8.2-beta with 3 brokers and 3 zookeeper nodes (3.4.6). I had some known issues with 0.8.1.1 that were supposedly fixed in 0.8.2, so I moved to 0.82-beta, but now I'm getting different errors in 0.8.2-beta. All the brokers come up fine and registered with the zookeeper. Then, I created a topic, which succeeds on the command line, but there are many errors in the kafka logs. Does anyone know why this may be happening? Thanks, Terry --- bin/kafka-topics.sh --create --zookeeper migration201:2181,migration202:2181,migration203:2181 --replication-factor 2 --topic nto1-mailboxes --partition 100 Created topic nto1-mailboxes. [2014-12-18 05:34:23,405] ERROR [ReplicaFetcherThread-0-202], Error for partition [nto1-mailboxes,29] to broker 202:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) [2014-12-18 05:34:23,406] ERROR [ReplicaFetcherThread-0-202], Error for partition [nto1-mailboxes,11] to broker 202:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) [2014-12-18 05:34:23,406] ERROR [ReplicaFetcherThread-0-202], Error for partition [nto1-mailboxes,23] to broker 202:class kafka.common.UnknownException (kafka.server.ReplicaFetcherThread) Another broker has: [2014-12-18 05:34:26,485] ERROR [KafkaApi-202] error when handling request Name: FetchRequest; Version: 0; CorrelationId : 463; ClientId: ReplicaFetcherThread-0-202; ReplicaId: 201; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [nto1-mail boxes,29] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,11] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,23] - P artitionFetchInfo(0,1048576),[nto1-mailboxes,83] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,5] - PartitionFetchIn fo(0,1048576),[nto1-mailboxes,59] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,17] - PartitionFetchInfo(0,1048576), [nto1-mailboxes,65] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,41] - PartitionFetchInfo(0,1048576),[nto1-mailboxe s,77] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,47] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,53] - Parti tionFetchInfo(0,1048576),[nto1-mailboxes,95] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,89] - PartitionFetchInfo( 0,1048576),[nto1-mailboxes,35] - PartitionFetchInfo(0,1048576),[nto1-mailboxes,71] - PartitionFetchInfo(0,1048576) (ka fka.server.KafkaApis) kafka.common.NotAssignedReplicaException: Leader 202 failed to record follower 201's position -1 since the replica is no t recognized to be one of the assigned replicas for partition [nto1-mailboxes,65] at kafka.server.ReplicaManager.updateReplicaLEOAndPartitionHW(ReplicaManager.scala:570) at kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:348) at kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:346) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at scala.collection.MapLike$MappedValues.foreach(MapLike.scala:245) at kafka.server.KafkaApis.recordFollowerLogEndOffsets(KafkaApis.scala:346) at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:311) at kafka.server.KafkaApis.handle(KafkaApis.scala:60) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) at java.lang.Thread.run(Unknown Source) However, the describe topic shows that everything is fine: Topic:nto1-mailboxesPartitionCount:100 ReplicationFactor:2 Configs: Topic: nto1-mailboxes Partition: 0Leader: 203 Replicas: 203,201 Isr: 203,201 Topic: nto1-mailboxes Partition: 1Leader: 201 Replicas: 201,202 Isr: 201,202 Topic: nto1-mailboxes Partition: 2Leader: 202 Replicas: 202,203 Isr: 202,203 Topic: nto1-mailboxes Partition: 3Leader: 203 Replicas: 203,202 Isr: 203,202 Topic: nto1-mailboxes Partition: 4Leader: 201 Replicas: 201,203 Isr: 201,203 Topic: nto1-mailboxes Partition: 5Leader: 202 Replicas: 202,201 Isr: 202,201 Topic: nto1-mailboxes Partition: 6Leader: 203 Replicas: 203,201 Isr: 203,201
Re: Kafka 0.8.2 new producer blocking on metadata
Yes, that's a potential issue. Perhaps we just need to have a lower default value for metadata.fetch.timeout.ms ? Thanks, Jun On Wed, Dec 17, 2014 at 11:10 PM, Paul Pearcy paul.pea...@blackboard.com wrote: Heya, Playing around with the 0.8.2-beta producer client. One of my test cases is to ensure producers can deal with Kafka being down when the producer is created. My tests failed miserably because of the default blocking in the producer with regard to metadata.fetch.timeout.ms. The first line of new producer is waitOnMetadata which is blocking. I can handle this case by loading topic meta on init and setting the timeout value to very low metadata.fetch.timeout.ms and either throwing away messages or creating my own internal queue to buffer. I'm surprised the metasync isn't done async. If it fails, return that in the future/callback. This way the API could actually be considered safely async and the producer buffer could try to hold on to things until block.on.buffer.full kicks in. You'd probably need a partition callback since numPartitions wouldn't be available. The implication is that people's apps will work fine if first messages are sent while kafka server is up, however, if kafka is down and they restart their app, the new producer will block all sends and blow things up if you haven't written your app to be aware of this edge case. Thanks, Paul This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
Re: Issues With Parallelism In Kafka Spout
Perhaps you can try getting some help from the Storm mailing list. Thanks, Jun On Thu, Dec 18, 2014 at 7:22 AM, Nilesh Chhapru nilesh.chha...@ugamsolutions.com wrote: Hi All, Please give some inputs as this is pending since long and need to meet the deadlines Regards, Nilesh Chhapru. From: Nilesh Chhapru [mailto:nilesh.chha...@ugamsolutions.com] Sent: 18 December 2014 01:24 PM To: u...@storm.apache.org; users@kafka.apache.org Subject: Issues With Parallelism In Kafka Spout Hi All, I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn't getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data. PFB the screen grab for number of task of spout and data emitted by only one of them. Please assist on how to achieve parallelism using high level Kafka spout. [cid:image001.png@01D01ABD.FA56FB90] Regards, Nilesh Chhapru. ---Disclaimer-- Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ---Disclaimer-- Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email.
Re: Kafka 0.8.2 new producer blocking on metadata
I double posted by accident, sorry. Have another thread discussing this. Thanks! On Dec 22, 2014 11:21 AM, Jun Rao j...@confluent.io wrote: Yes, that's a potential issue. Perhaps we just need to have a lower default value for metadata.fetch.timeout.ms ? Thanks, Jun On Wed, Dec 17, 2014 at 11:10 PM, Paul Pearcy paul.pea...@blackboard.com wrote: Heya, Playing around with the 0.8.2-beta producer client. One of my test cases is to ensure producers can deal with Kafka being down when the producer is created. My tests failed miserably because of the default blocking in the producer with regard to metadata.fetch.timeout.ms. The first line of new producer is waitOnMetadata which is blocking. I can handle this case by loading topic meta on init and setting the timeout value to very low metadata.fetch.timeout.ms and either throwing away messages or creating my own internal queue to buffer. I'm surprised the metasync isn't done async. If it fails, return that in the future/callback. This way the API could actually be considered safely async and the producer buffer could try to hold on to things until block.on.buffer.full kicks in. You'd probably need a partition callback since numPartitions wouldn't be available. The implication is that people's apps will work fine if first messages are sent while kafka server is up, however, if kafka is down and they restart their app, the new producer will block all sends and blow things up if you haven't written your app to be aware of this edge case. Thanks, Paul This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
Re: Increase in Kafka replication fetcher thread not reducing log replication
Increasing replica.fetch.max.bytes will help, but will only get diminishing returns at some point. Increasing num.replica.fetchers will also help. You need to make sure that the leaders are balanced in the cluster. See http://kafka.apache.org/documentation.html#basic_ops_leader_balancing Thanks, Jun On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi All, I am trying to figure out best configuration for my Kafka brokers so that in case of restarted, the new node catch up with Leader at quick pace. My test environment has 2 kafka brokers and 1 Topic with one Partition. I first ran the test (Test#1) with default setting, i.e. num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB). it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( 37.9*1024/700) Later I increased the num.replica.fetchers = 5 and replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec. i ran two more test and results got much worse: Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. replication rate = 92.7 MB/sec Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. replication rate = 93.54 MB/sec Any reason why increasing the replica fetcher thread or increase in fetch max bytes not increasing my replication rate linearly. note: in all the test CPU utilization was not more than 45% Regards, Nitin Kumar Sharma.
Re: Increase in Kafka replication fetcher thread not reducing log replication
What kind of network do you have? gigabit? if so 90 MB/s would make sense Also since you have one partition what's your raw transfer speed to the disk? 90 MB/s makes sense here as well... If I were looking for rapid replica catch up I'd have at least 2x Gbit and partitioned topics spread out over several physical nodes. That would make it possible to catch up at ~200 MB/s without totally blocking normal operation of the cluster. For a single partition you would face lower catch up rate since all data is coming from one physical disk /svante 2014-12-22 17:26 GMT+01:00 Jun Rao j...@confluent.io: Increasing replica.fetch.max.bytes will help, but will only get diminishing returns at some point. Increasing num.replica.fetchers will also help. You need to make sure that the leaders are balanced in the cluster. See http://kafka.apache.org/documentation.html#basic_ops_leader_balancing Thanks, Jun On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi All, I am trying to figure out best configuration for my Kafka brokers so that in case of restarted, the new node catch up with Leader at quick pace. My test environment has 2 kafka brokers and 1 Topic with one Partition. I first ran the test (Test#1) with default setting, i.e. num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB). it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( 37.9*1024/700) Later I increased the num.replica.fetchers = 5 and replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec. i ran two more test and results got much worse: Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. replication rate = 92.7 MB/sec Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. replication rate = 93.54 MB/sec Any reason why increasing the replica fetcher thread or increase in fetch max bytes not increasing my replication rate linearly. note: in all the test CPU utilization was not more than 45% Regards, Nitin Kumar Sharma.
Re: kafka-producer-perf-test.sh
If that kafka-producer-per-test.sh doesn't work for you and you choose the DIY approach, https://github.com/sematext/ActionGenerator may be of help. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Mon, Dec 22, 2014 at 11:09 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: have to conduce load and stress test by publishing at least 10K messages/sec to my Kafka brokers. Here is the setup: Number of brokers : 2 Topic : 1 Partition :2. Each broker leader of one. I would like to know if i can use kafka-producer-per-test.sh in first place. also, if any one knows the limitation of this utility, so that i can think of writing my own in order to overcome the limitations. Regards, Nitin Kumar Sharma. On Mon, Dec 22, 2014 at 10:57 AM, Jun Rao j...@confluent.io wrote: What kind of load testing do you plan to do? Thanks, Jun On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi Team, I have to make a decision on whether i should go with Kafka producer test utility or build my own java tool for my load testing . Kindly let me know if anyone knows any limitation with kafka-producer-per-test.sh when it come to simulation of messages under load condition? Regards, Nitin Kumar Sharma.
RE: Kafka in C#
Hi there, We looked at both of these a while back and ended up writing our own (https://github.com/ntent-ad/kafka4net). The first one in the list was completely synchronous, and had no concept of batching. We initially attempted to use the second one (kafka-net) but had some issues with detecting leader changes, temporary errors, and general topic metadata changes. Also had some problems with the use of async and the tracking of correlation IDs on tcp messages. I know our client does not have a lot of documentation (yet) and says Work in progress, not ready yet! but we have been testing this a lot, and have been running in staging under load with good results. We will be entering production in the next few weeks (waiting until after the holidays). I wouldn't be comfortable with it going straight to production without some testing in your environment, but especially the Producer is very robust, and is resilient to all sorts of changes in the environment (the integration tests use Vagrant and a set of VMs and test Producing to non-existent partition and waiting for auto-creation of topic, partition rebalancing, broker down or other partition re-assignment, etc.) The client is fully Async and leverages Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all processing. Changes in partition state are broadcast Rx style to listening components (Producer, Consumer, PartitionRecoveryMonitor). One of the reasons we have not finalized the documentation and notice on the github page is we weren't sure if the API might change based on usage. To this end, we'd like to know what you think and if you have any use-cases not handled by the API. I understand if you're not comfortable with the beta state of the client, but we'd love to have you check it out. We are actively developing on this and can help with any issues. Thanks, Thunder -Original Message- From: Matti Waarna [mailto:mwaa...@sapient.com] Sent: Monday, December 22, 2014 7:55 AM To: users@kafka.apache.org Subject: Kafka in C# We are using kafka version 0.8.1.1 and trying to produce from a C# app. I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available. I am looking for a solution that is a) stable enough for production environment and b) performs well. 1) A couple of active github projects are available along with a few forks each. Has Anybody worked on either of the following two options to contribute their findings? https://github.com/Jroland/kafka-net https://github.com/miknil/Kafka4n 2) Also there is the option of IKVM to import kafka JARS into a .net DLL. Maybe even another solution? Thanks MATTI
Re: Kafka in C#
Thunder, can you add that to https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net didn't know it existed but cool that it uses Rx On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges tstump...@ntent.com wrote: Hi there, We looked at both of these a while back and ended up writing our own ( https://github.com/ntent-ad/kafka4net). The first one in the list was completely synchronous, and had no concept of batching. We initially attempted to use the second one (kafka-net) but had some issues with detecting leader changes, temporary errors, and general topic metadata changes. Also had some problems with the use of async and the tracking of correlation IDs on tcp messages. I know our client does not have a lot of documentation (yet) and says Work in progress, not ready yet! but we have been testing this a lot, and have been running in staging under load with good results. We will be entering production in the next few weeks (waiting until after the holidays). I wouldn't be comfortable with it going straight to production without some testing in your environment, but especially the Producer is very robust, and is resilient to all sorts of changes in the environment (the integration tests use Vagrant and a set of VMs and test Producing to non-existent partition and waiting for auto-creation of topic, partition rebalancing, broker down or other partition re-assignment, etc.) The client is fully Async and leverages Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all processing. Changes in partition state are broadcast Rx style to listening components (Producer, Consumer, PartitionRecoveryMonitor). One of the reasons we have not finalized the documentation and notice on the github page is we weren't sure if the API might change based on usage. To this end, we'd like to know what you think and if you have any use-cases not handled by the API. I understand if you're not comfortable with the beta state of the client, but we'd love to have you check it out. We are actively developing on this and can help with any issues. Thanks, Thunder -Original Message- From: Matti Waarna [mailto:mwaa...@sapient.com] Sent: Monday, December 22, 2014 7:55 AM To: users@kafka.apache.org Subject: Kafka in C# We are using kafka version 0.8.1.1 and trying to produce from a C# app. I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available. I am looking for a solution that is a) stable enough for production environment and b) performs well. 1) A couple of active github projects are available along with a few forks each. Has Anybody worked on either of the following two options to contribute their findings? https://github.com/Jroland/kafka-net https://github.com/miknil/Kafka4n 2) Also there is the option of IKVM to import kafka JARS into a .net DLL. Maybe even another solution? Thanks MATTI
RE: Kafka in C#
We definitely will. Wanted to wait until we got more confident the API was solid, ran in production for a little, and added a bit more documentation. Cheers, Thunder -Original Message- From: Joe Stein [joe.st...@stealth.ly] Received: Monday, 22 Dec 2014, 10:05AM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Kafka in C# Thunder, can you add that to https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net didn't know it existed but cool that it uses Rx On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges tstump...@ntent.com wrote: Hi there, We looked at both of these a while back and ended up writing our own ( https://github.com/ntent-ad/kafka4net). The first one in the list was completely synchronous, and had no concept of batching. We initially attempted to use the second one (kafka-net) but had some issues with detecting leader changes, temporary errors, and general topic metadata changes. Also had some problems with the use of async and the tracking of correlation IDs on tcp messages. I know our client does not have a lot of documentation (yet) and says Work in progress, not ready yet! but we have been testing this a lot, and have been running in staging under load with good results. We will be entering production in the next few weeks (waiting until after the holidays). I wouldn't be comfortable with it going straight to production without some testing in your environment, but especially the Producer is very robust, and is resilient to all sorts of changes in the environment (the integration tests use Vagrant and a set of VMs and test Producing to non-existent partition and waiting for auto-creation of topic, partition rebalancing, broker down or other partition re-assignment, etc.) The client is fully Async and leverages Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all processing. Changes in partition state are broadcast Rx style to listening components (Producer, Consumer, PartitionRecoveryMonitor). One of the reasons we have not finalized the documentation and notice on the github page is we weren't sure if the API might change based on usage. To this end, we'd like to know what you think and if you have any use-cases not handled by the API. I understand if you're not comfortable with the beta state of the client, but we'd love to have you check it out. We are actively developing on this and can help with any issues. Thanks, Thunder -Original Message- From: Matti Waarna [mailto:mwaa...@sapient.com] Sent: Monday, December 22, 2014 7:55 AM To: users@kafka.apache.org Subject: Kafka in C# We are using kafka version 0.8.1.1 and trying to produce from a C# app. I realize that there is no official C# library release and want to get your experience with the existing solutions that are currently available. I am looking for a solution that is a) stable enough for production environment and b) performs well. 1) A couple of active github projects are available along with a few forks each. Has Anybody worked on either of the following two options to contribute their findings? https://github.com/Jroland/kafka-net https://github.com/miknil/Kafka4n 2) Also there is the option of IKVM to import kafka JARS into a .net DLL. Maybe even another solution? Thanks MATTI
the impact of partition number
Hi, All I've run bin/kafka-producer-perf-test.sh on our kafka-production cluster, I found the number of partitions really have huge impacts on the producer performance, see: start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2014-12-22 19:53:27:392, 2014-12-22 19:54:25:581, 1, 3000, 200, 2861.02, 49.1678, 100, 17185.3787 2014-12-22 19:55:27:048, 2014-12-22 19:56:23:318, 1, 3000, 200, 2861.02, 50.8446, 100, 17771.4590 2014-12-22 19:58:09:466, 2014-12-22 19:59:05:068, 1, 3000, 200, 2861.02, 51.4554, 100, 17984.9646 2014-12-22 19:59:40:389, 2014-12-22 20:00:28:646, 1, 3000, 200, 2861.02, 59.2872, 100, 20722.3822 2014-12-22 20:02:41:993, 2014-12-22 20:03:22:481, 1, 3000, 200, 2861.02, 70.6635, 100, 24698.6762 2014-12-22 20:03:47:594, 2014-12-22 20:04:26:238, 1, 3000, 200, 2861.02, 74.0354, 100, 25877.2384 2014-12-22 20:11:49:492, 2014-12-22 20:12:25:843, 1, 3000, 200, 2861.02, 78.7055, 100, 27509.5596 2014-12-22 20:12:53:290, 2014-12-22 20:13:29:746, 1, 3000, 200, 2861.02, 78.4788, 100, 27430.3270 2014-12-22 20:13:53:194, 2014-12-22 20:14:29:470, 1, 3000, 200, 2861.02, 78.8682, 100, 27566.4351 2014-12-22 20:14:51:491, 2014-12-22 20:15:25:451, 1, 3000, 200, 2861.02, 84.2468, 100, 29446.4075 2014-12-22 20:16:51:369, 2014-12-22 20:17:27:452, 1, 3000, 200, 2861.02, 79.2901, 100, 27713.8819 2014-12-22 20:17:57:882, 2014-12-22 20:18:33:957, 1, 3000, 200, 2861.02, 79.3076, 100, 27720.0277 The number of partitions above are from 1 to 12, I wonder why it has such big difference? thanks -- Alec Li
leader and isr were not set when create the topic
Hi, All I created a topic with 3 replications and 6 partitions, but when I check this topic, seems there is no leader and isr were set for this topic, see bin/kafka-topics.sh --create --zookeeper 10.100.98.100:2181 --replication-factor 3 --partitions 6 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Created topic perf_producer_p6_test. root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p6_test PartitionCount:6 ReplicationFactor:3 Configs: Topic: perf_producer_p6_testPartition: 0Leader: none Replicas: 100,101,102 Isr: Topic: perf_producer_p6_testPartition: 1Leader: none Replicas: 101,102,100 Isr: Topic: perf_producer_p6_testPartition: 2Leader: none Replicas: 102,100,101 Isr: Topic: perf_producer_p6_testPartition: 3Leader: none Replicas: 100,102,101 Isr: Topic: perf_producer_p6_testPartition: 4Leader: none Replicas: 101,100,102 Isr: Topic: perf_producer_p6_testPartition: 5Leader: none Replicas: 102,101,100 Isr: Is there a way to specifically set leader and isr in command line, it is strange when I create the topic with 5 partitions, it has leader and isr: root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p5_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p5_test PartitionCount:5 ReplicationFactor:3 Configs: Topic: perf_producer_p5_testPartition: 0Leader: 102 Replicas: 102,100,101 Isr: 102,100,101 Topic: perf_producer_p5_testPartition: 1Leader: 102 Replicas: 100,101,102 Isr: 102,101 Topic: perf_producer_p5_testPartition: 2Leader: 101 Replicas: 101,102,100 Isr: 101,102,100 Topic: perf_producer_p5_testPartition: 3Leader: 102 Replicas: 102,101,100 Isr: 102,101,100 Topic: perf_producer_p5_testPartition: 4Leader: 102 Replicas: 100,102,101 Isr: 102,101 Any ideas? thanks -- Alec Li
Re: Kafka consumer session timeouts
Terry, The zookeeper client used by the high level Kafka consumer has a separate thread that does the heartbeat in the background. So even if it takes long to process the message, it should not make the consumer's session to time out or make the consumer rebalance. You may be running into long GC pauses on your consumer that might be causing the session timeouts. Thanks, Neha On Fri, Dec 19, 2014 at 12:51 PM, Terry Cumaranatunge cumar...@gmail.com wrote: Hi I would like to get some feedback on design choices with kafka consumers. We have an application that a consumer reads a message and the thread does a number of things, including database accesses before a message is produced to another topic. The time between consuming and producing the message on the thread can take several minutes. Once message is produced to new topic, a commit is done to indicate we are done with work on the consumer queue message. Auto commit is disabled for this reason. I'm using the high level consumer and what I'm noticing is that zookeeper and kafka sessions timeout because it is taking too long before we do anything on consumer queue so kafka ends up rebalancing every time the thread goes back to read more from consumer queue and it starts to take a long time before a consumer reads a new message after a while. I can set zookeeper session timeout very high to not make that a problem but then i have to adjust the rebalance parameters accordingly and kafka won't pickup a new consumer for a while among other side effects. What are my options to solve this problem? Is there a way to heartbeat to kafka and zookeeper to keep both happy? Do i still have these same issues if i were to use a simple consumer? Thanks -- Thanks, Neha
Re: leader and isr were not set when create the topic
There is possibly some error in your broker logs. Can you check if you see any and send it around? On Mon, Dec 22, 2014 at 2:46 PM, Sa Li sal...@gmail.com wrote: Hi, All I created a topic with 3 replications and 6 partitions, but when I check this topic, seems there is no leader and isr were set for this topic, see bin/kafka-topics.sh --create --zookeeper 10.100.98.100:2181 --replication-factor 3 --partitions 6 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Created topic perf_producer_p6_test. root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p6_test PartitionCount:6 ReplicationFactor:3 Configs: Topic: perf_producer_p6_testPartition: 0Leader: none Replicas: 100,101,102 Isr: Topic: perf_producer_p6_testPartition: 1Leader: none Replicas: 101,102,100 Isr: Topic: perf_producer_p6_testPartition: 2Leader: none Replicas: 102,100,101 Isr: Topic: perf_producer_p6_testPartition: 3Leader: none Replicas: 100,102,101 Isr: Topic: perf_producer_p6_testPartition: 4Leader: none Replicas: 101,100,102 Isr: Topic: perf_producer_p6_testPartition: 5Leader: none Replicas: 102,101,100 Isr: Is there a way to specifically set leader and isr in command line, it is strange when I create the topic with 5 partitions, it has leader and isr: root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p5_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p5_test PartitionCount:5 ReplicationFactor:3 Configs: Topic: perf_producer_p5_testPartition: 0Leader: 102 Replicas: 102,100,101 Isr: 102,100,101 Topic: perf_producer_p5_testPartition: 1Leader: 102 Replicas: 100,101,102 Isr: 102,101 Topic: perf_producer_p5_testPartition: 2Leader: 101 Replicas: 101,102,100 Isr: 101,102,100 Topic: perf_producer_p5_testPartition: 3Leader: 102 Replicas: 102,101,100 Isr: 102,101,100 Topic: perf_producer_p5_testPartition: 4Leader: 102 Replicas: 100,102,101 Isr: 102,101 Any ideas? thanks -- Alec Li -- Thanks, Neha
Re: leader and isr were not set when create the topic
I restart the kafka server, it is the same thing, sometime nothing listed on ISR, leader, I checked the state-change log [2014-12-22 23:46:38,164] TRACE Broker 100 cached leader info (LeaderAndIsrInfo:(Leader:101,ISR:101,102,100,LeaderEpoch:0,ControllerEpoch:4),ReplicationFactor:3),AllReplicas:101,102,100) for partition [perf_producer_p8_test,1] in response to UpdateMetadata request sent by controller 101 epoch 4 with correlation id 138 (state.change.logger) On Mon, Dec 22, 2014 at 2:46 PM, Sa Li sal...@gmail.com wrote: Hi, All I created a topic with 3 replications and 6 partitions, but when I check this topic, seems there is no leader and isr were set for this topic, see bin/kafka-topics.sh --create --zookeeper 10.100.98.100:2181 --replication-factor 3 --partitions 6 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Created topic perf_producer_p6_test. root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p6_test PartitionCount:6 ReplicationFactor:3 Configs: Topic: perf_producer_p6_testPartition: 0Leader: none Replicas: 100,101,102 Isr: Topic: perf_producer_p6_testPartition: 1Leader: none Replicas: 101,102,100 Isr: Topic: perf_producer_p6_testPartition: 2Leader: none Replicas: 102,100,101 Isr: Topic: perf_producer_p6_testPartition: 3Leader: none Replicas: 100,102,101 Isr: Topic: perf_producer_p6_testPartition: 4Leader: none Replicas: 101,100,102 Isr: Topic: perf_producer_p6_testPartition: 5Leader: none Replicas: 102,101,100 Isr: Is there a way to specifically set leader and isr in command line, it is strange when I create the topic with 5 partitions, it has leader and isr: root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p5_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p5_test PartitionCount:5 ReplicationFactor:3 Configs: Topic: perf_producer_p5_testPartition: 0Leader: 102 Replicas: 102,100,101 Isr: 102,100,101 Topic: perf_producer_p5_testPartition: 1Leader: 102 Replicas: 100,101,102 Isr: 102,101 Topic: perf_producer_p5_testPartition: 2Leader: 101 Replicas: 101,102,100 Isr: 101,102,100 Topic: perf_producer_p5_testPartition: 3Leader: 102 Replicas: 102,101,100 Isr: 102,101,100 Topic: perf_producer_p5_testPartition: 4Leader: 102 Replicas: 100,102,101 Isr: 102,101 Any ideas? thanks -- Alec Li -- Alec Li
Re: Rebalance not happening even after increasing max retries causing conflict in ZK
Can you share a reproducible test case?
On Tue, Dec 9, 2014 at 7:11 AM, Mohit Kathuria mkathu...@sprinklr.com
wrote:
Neha,
The same issue reoccured with just 2 consumer processes. The exception was
related to conflict in writing the ephemeral node. Below was the exception.
Topic name is
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin
with 30 partitions. The 2 processes were running on 2 servers with ips
10.0.8.222 and 10.0.8.225.
*2014-12-09 13:22:11 k.u.ZkUtils$ [INFO] I wrote this conflicted ephemeral
node
[{version:1,subscription:{lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin:5},pattern:static,timestamp:1417964160024}]
at
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
a while back in a different session, hence I will backoff for this node to
be deleted by Zookeeper and retry*
Attached the complete error logs. The exception occured after the
rebalance failed even after 40 retries. Rebalance failed as the process
already owning some of the partitions did not give us ownership due to
conflicting ephemeral nodes. As you suggested, we ran the wchp command on
the 3 zookeeper nodes at this time and figured out that the watcher was
registered for only one of the process. I am copying the kafka consumer
watcher registered on one of the zookeeper servers. (Attached are the wchp
outputs of all 3 zk servers)
*$echo wchp | nc localhost 2181 *
*/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids*
* 0x34a175e1d5d0130*
0x34a175e1d5d0130 was the ephemeral node session Id. I went back to the
zookeeper shell and checked the consumers registered for this topic and
consumer group(same as topic name). Attaching the output in zkCommands.txt.
This clearly shows that
10.0.8.222 has ephemeralOwner = 0x34a175e1d5d0130
10.0.8.225 has ephemeralOwner = 0x34a175e1d5d0127
I think we have the issue here that both consumers have written to
different ephemeral nodes. Watchers are registered for the one of the 2
ephemeral node. The root cause seems to be the inconsistent state while
writing the ephemeral nodes in ZK.
Let me know if you need more details.
-Thanks,
Mohit
On Mon, Nov 10, 2014 at 8:46 AM, Neha Narkhede neha.narkh...@gmail.com
wrote:
A rebalance should trigger on all consumers when you add a new consumer to
the group. If you don't see the zookeeper watch fire, the consumer may
have
somehow lost the watch. We have seen this behavior on older zk versions, I
wonder it that bug got reintroduced. To verify if this is the case, you
can
run the wchp zookeeper command on the zk leader and check if each consumer
has a watch registered.
Do you have a way to try this on zk 3.3.4? I would recommend you try the
wchp suggestion as well.
On Fri, Nov 7, 2014 at 6:07 AM, Mohit Kathuria mkathu...@sprinklr.com
wrote:
Hi all,
Can someone help here. We are getting constant rebalance failure each
time
a consumer is added beyond a certain number. Did quite a lot of
debugging
on this and still not able to figure out the pattern.
-Thanks,
Mohit
On Mon, Nov 3, 2014 at 10:53 PM, Mohit Kathuria mkathu...@sprinklr.com
wrote:
Neha,
Looks like an issue with the consumer rebalance not able to complete
successfully. We were able to reproduce the issue on topic with 30
partitions, 3 consumer processes(p1,p2 and p3), properties - 40
rebalance.max.retries and 1(10s) rebalance.backoff.ms.
Before the process p3 was started, partition ownership was as
expected:
partitions 0-14 owned by p1
partitions 15-29 - owner p2
As the process p3 started, rebalance was triggered. Process p3 was
successfully able to acquire partition ownership for partitions 20-29
as
expected as per the rebalance algorithm. However, process p2 while
trying
to acquire ownership of partitions 10-19 saw rebalance failure after
40
retries.
Attaching the logs from process p2 and process p1. It says that p2 was
attempting to rebalance, it was trying to acquire ownership of
partitions
10-14 which were owned by process p1. However, at the same time
process
p1
did not get any event for giving up the partition ownership for
partitions
1-14.
We were expecting a rebalance to have triggered in p1 - but it didn't
and
hence not giving up ownership. Is our assumption correct/incorrect?
And if the rebalance gets triggered in p1 - how to figure out apart
from
logs as the logs on p1 did not have anything.
*2014-11-03 06:57:36 k.c.ZookeeperConsumerConnector [INFO]
[topic_consumerIdString], waiting for the partition ownership to be
deleted: 11*
During and after the rebalance failed on process p2, Partition
Ownership
was as below:
0-14 -
Re: leader and isr were not set when create the topic
Hello, Neha This is the error from server.log [2014-12-22 23:53:25,663] WARN [KafkaApi-100] Fetch request with correlation id 1227732 from client ReplicaFetcherThread-0-100 on partition [perf_producer_p8_test,1] failed due to Leader not local for partition [perf_producer_p8_test,1] on broker 100 (kafka.server.KafkaApis) On Mon, Dec 22, 2014 at 3:50 PM, Sa Li sal...@gmail.com wrote: I restart the kafka server, it is the same thing, sometime nothing listed on ISR, leader, I checked the state-change log [2014-12-22 23:46:38,164] TRACE Broker 100 cached leader info (LeaderAndIsrInfo:(Leader:101,ISR:101,102,100,LeaderEpoch:0,ControllerEpoch:4),ReplicationFactor:3),AllReplicas:101,102,100) for partition [perf_producer_p8_test,1] in response to UpdateMetadata request sent by controller 101 epoch 4 with correlation id 138 (state.change.logger) On Mon, Dec 22, 2014 at 2:46 PM, Sa Li sal...@gmail.com wrote: Hi, All I created a topic with 3 replications and 6 partitions, but when I check this topic, seems there is no leader and isr were set for this topic, see bin/kafka-topics.sh --create --zookeeper 10.100.98.100:2181 --replication-factor 3 --partitions 6 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Created topic perf_producer_p6_test. root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p6_test PartitionCount:6 ReplicationFactor:3 Configs: Topic: perf_producer_p6_testPartition: 0Leader: none Replicas: 100,101,102 Isr: Topic: perf_producer_p6_testPartition: 1Leader: none Replicas: 101,102,100 Isr: Topic: perf_producer_p6_testPartition: 2Leader: none Replicas: 102,100,101 Isr: Topic: perf_producer_p6_testPartition: 3Leader: none Replicas: 100,102,101 Isr: Topic: perf_producer_p6_testPartition: 4Leader: none Replicas: 101,100,102 Isr: Topic: perf_producer_p6_testPartition: 5Leader: none Replicas: 102,101,100 Isr: Is there a way to specifically set leader and isr in command line, it is strange when I create the topic with 5 partitions, it has leader and isr: root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p5_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p5_test PartitionCount:5 ReplicationFactor:3 Configs: Topic: perf_producer_p5_testPartition: 0Leader: 102 Replicas: 102,100,101 Isr: 102,100,101 Topic: perf_producer_p5_testPartition: 1Leader: 102 Replicas: 100,101,102 Isr: 102,101 Topic: perf_producer_p5_testPartition: 2Leader: 101 Replicas: 101,102,100 Isr: 101,102,100 Topic: perf_producer_p5_testPartition: 3Leader: 102 Replicas: 102,101,100 Isr: 102,101,100 Topic: perf_producer_p5_testPartition: 4Leader: 102 Replicas: 100,102,101 Isr: 102,101 Any ideas? thanks -- Alec Li -- Alec Li -- Alec Li
Re: leader and isr were not set when create the topic
I have three nodes: 100, 101, and 102 When I restart all of them, seems now everything is ok, but I would like to paste the error messages I got from server.log from each node, see if you can help to understand what is the problem. on node 100 [2014-12-23 00:04:39,401] ERROR [KafkaApi-100] Error when processing fetch request for partition [perf_producer_p8_test,7] offset 125000 from follower with correlation id 0 (kafka.server.KafkaApis) kafka.common.OffsetOutOfRangeException: Request for offset 125000 but we only have log segments in the range 0 to 0. at kafka.log.Log.read(Log.scala:380) at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:530) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:476) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:471) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map3.foreach(Map.scala:154) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) .. .. in Node 101 and 102 [2014-12-23 00:04:39,440] ERROR [ReplicaFetcherThread-0-100], Current offset 1 25000 for partition [perf_producer_p8_test,1] out of range; reset offset to 0 (kafka.server.ReplicaFetcherThread) [2014-12-23 00:04:39,442] INFO Truncating log perf_producer_p8_test-7 to offset 0. (kafka.log.Log) [2014-12-23 00:04:39,452] WARN [ReplicaFetcherThread-0-100], Replica 102 for partition [perf_producer_p8_test,7] reset its fetch offset to current leader 100's latest offset 0 (kafka.server.ReplicaFetcherThread) On Mon, Dec 22, 2014 at 3:55 PM, Sa Li sal...@gmail.com wrote: Hello, Neha This is the error from server.log [2014-12-22 23:53:25,663] WARN [KafkaApi-100] Fetch request with correlation id 1227732 from client ReplicaFetcherThread-0-100 on partition [perf_producer_p8_test,1] failed due to Leader not local for partition [perf_producer_p8_test,1] on broker 100 (kafka.server.KafkaApis) On Mon, Dec 22, 2014 at 3:50 PM, Sa Li sal...@gmail.com wrote: I restart the kafka server, it is the same thing, sometime nothing listed on ISR, leader, I checked the state-change log [2014-12-22 23:46:38,164] TRACE Broker 100 cached leader info (LeaderAndIsrInfo:(Leader:101,ISR:101,102,100,LeaderEpoch:0,ControllerEpoch:4),ReplicationFactor:3),AllReplicas:101,102,100) for partition [perf_producer_p8_test,1] in response to UpdateMetadata request sent by controller 101 epoch 4 with correlation id 138 (state.change.logger) On Mon, Dec 22, 2014 at 2:46 PM, Sa Li sal...@gmail.com wrote: Hi, All I created a topic with 3 replications and 6 partitions, but when I check this topic, seems there is no leader and isr were set for this topic, see bin/kafka-topics.sh --create --zookeeper 10.100.98.100:2181 --replication-factor 3 --partitions 6 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Created topic perf_producer_p6_test. root@precise64:/etc/kafka# bin/kafka-topics.sh --describe --zookeeper 10.100.98.100:2181 --topic perf_producer_p6_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/etc/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Topic:perf_producer_p6_test PartitionCount:6 ReplicationFactor:3 Configs: Topic: perf_producer_p6_testPartition: 0Leader: none Replicas: 100,101,102 Isr: Topic: perf_producer_p6_testPartition: 1Leader: none Replicas: 101,102,100 Isr: Topic: perf_producer_p6_testPartition: 2Leader: none Replicas: 102,100,101 Isr: Topic: perf_producer_p6_testPartition: 3Leader: none Replicas: 100,102,101 Isr: Topic: perf_producer_p6_test
kafka monitoring system
Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your monitoring system. The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all browsing all metrics with JMX. We pay particular we do graphing and alerting on the following metrics: .. I am wondering if anyone ever use Jconsole to monitor the kafka, or anyone can recommend a good monitoring tool for kafka production. thanks -- Alec Li
Kafka 0.8.1.1 eadership changes are happening very often
Hello, I am running 3 brokers, one zookeeper and producer all on separate machine. I am also sending very low load around 6K msg/sec. Each msg is around 150 bytes only. I ran the load for only 5 minutes and during this time, I see leadership chained very often. I created 3 partitions. Here leadership for each partitions changed. All 3 brokers are running perfectly fine. No broker is down. Could someone let me know why kafka leadership changed very often. Initially: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 2Replicas: 2,3,1 Isr: 2,3,1 Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,2,3 Changed to: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 3Replicas: 2,3,1 Isr: 3,1,2 Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,3,2 Changed to: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 1Replicas: 2,3,1 Isr: 1,2,3 Topic: mmetopic1Partition: 1 Leader: 1Replicas: 3,1,2 Isr: 1,2,3 Topic: mmetopic1Partition: 2 Leader: 2Replicas: 1,2,3 Isr: 2,1,3 Changed to: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 3Replicas: 2,3,1 Isr: 3,1,2 Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,3,2 Thanks, Lokesh
RE: Kafka 0.8.1.1 eadership changes are happening very often
Did you check the GC logs in the server? We ran into this and the default setting of 1G max heap on the broker process was nowhere near enough. We currently have set to 4G. -T -Original Message- From: Birla, Lokesh [lokesh.bi...@verizon.com] Received: Monday, 22 Dec 2014, 5:27PM To: users@kafka.apache.org [users@kafka.apache.org] CC: Birla, Lokesh [lokesh.bi...@verizon.com] Subject: Kafka 0.8.1.1 eadership changes are happening very often Hello, I am running 3 brokers, one zookeeper and producer all on separate machine. I am also sending very low load around 6K msg/sec. Each msg is around 150 bytes only. I ran the load for only 5 minutes and during this time, I see leadership chained very often. I created 3 partitions. Here leadership for each partitions changed. All 3 brokers are running perfectly fine. No broker is down. Could someone let me know why kafka leadership changed very often. Initially: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 2Replicas: 2,3,1 Isr: 2,3,1 Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,2,3 Changed to: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 3Replicas: 2,3,1 Isr: 3,1,2 Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,3,2 Changed to: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 1Replicas: 2,3,1 Isr: 1,2,3 Topic: mmetopic1Partition: 1 Leader: 1Replicas: 3,1,2 Isr: 1,2,3 Topic: mmetopic1Partition: 2 Leader: 2Replicas: 1,2,3 Isr: 2,1,3 Changed to: Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: Topic: mmetopic1Partition: 0 Leader: 3Replicas: 2,3,1 Isr: 3,1,2 Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,3,2 Thanks, Lokesh
Re: kafka monitoring system
Hi Sa Li, Have a look at SPM for monitoring Kafka: http://sematext.com/spm http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/ https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+FAQ#SPMFAQ-KafkaMonitoring Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Mon, Dec 22, 2014 at 7:41 PM, Sa Li sal...@gmail.com wrote: Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your monitoring system. The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all browsing all metrics with JMX. We pay particular we do graphing and alerting on the following metrics: .. I am wondering if anyone ever use Jconsole to monitor the kafka, or anyone can recommend a good monitoring tool for kafka production. thanks -- Alec Li
Re: kafka monitoring system
Hi Sa Li, You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is collecting data with JMX and sending to opentsdb. Opentsdb is graphing and alerting. YuanJia Li From: Sa Li Date: 2014-12-23 08:41 To: users Subject: kafka monitoring system Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your monitoring system. The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all browsing all metrics with JMX. We pay particular we do graphing and alerting on the following metrics: .. I am wondering if anyone ever use Jconsole to monitor the kafka, or anyone can recommend a good monitoring tool for kafka production. thanks -- Alec Li
Re: kafka monitoring system
Hi Sa Li, You can also try jmxtrans + graphite (for charting). jmxtrans has graphite output adapter out of the box. Regards, Raja. On Mon, Dec 22, 2014 at 10:39 PM, YuanJia Li yuanjia8...@163.com wrote: Hi Sa Li, You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is collecting data with JMX and sending to opentsdb. Opentsdb is graphing and alerting. YuanJia Li From: Sa Li Date: 2014-12-23 08:41 To: users Subject: kafka monitoring system Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your monitoring system. The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all browsing all metrics with JMX. We pay particular we do graphing and alerting on the following metrics: .. I am wondering if anyone ever use Jconsole to monitor the kafka, or anyone can recommend a good monitoring tool for kafka production. thanks -- Alec Li -- Thanks, Raja.
Re: kafka monitoring system
Try SemaText : a fully managed monitoring SaaS : http://sematext.com/ ... On Mon, Dec 22, 2014 at 7:46 PM, Rajasekar Elango rela...@salesforce.com wrote: Hi Sa Li, You can also try jmxtrans + graphite (for charting). jmxtrans has graphite output adapter out of the box. Regards, Raja. On Mon, Dec 22, 2014 at 10:39 PM, YuanJia Li yuanjia8...@163.com wrote: Hi Sa Li, You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is collecting data with JMX and sending to opentsdb. Opentsdb is graphing and alerting. YuanJia Li From: Sa Li Date: 2014-12-23 08:41 To: users Subject: kafka monitoring system Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your monitoring system. The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all browsing all metrics with JMX. We pay particular we do graphing and alerting on the following metrics: .. I am wondering if anyone ever use Jconsole to monitor the kafka, or anyone can recommend a good monitoring tool for kafka production. thanks -- Alec Li -- Thanks, Raja.
