how many instances of the broker is suggested in one server.
Hi all, I have one question, how many instances of the broker is suggested in one server? We start three instances, the performance will be three times faster than one instance. So how many instances should I start? Most appreciated zhujie
Re: 答复: kafka performance question
Do you read from the file in the callback from kafka? I just implemented c++ bindings and in one of the tests i did I got the following results: 1000 messages per batch (fairly small messages ~150 bytes) and then wait for the network layer to ack the send (not server ack)'s before putting another message on the tcp socket. This seems to give me a average latency of 17 ms. Througput about 10MB/s . If you are serializing your requests and is reading data from disk between calls to kafka then that would easily explain some added milliseconds in each call and thus a reduced throughput. Partitioning will not reduce latency. /svante 2014-05-26 6:18 GMT+02:00 Zhujie (zhujie, Smartcare) first.zhu...@huawei.com: Only one broker，and eight partitions, async mode. Increase the number of batch.num.messages is useless. We split the whole file into 1K per block. -邮件原件- 发件人: robairrob...@gmail.com [mailto:robairrob...@gmail.com] 代表 Robert Turner 发送时间: 2014年5月16日 13:45 收件人: users@kafka.apache.org 主题: Re: kafka performance question A couple of thoughts spring to mind, are you sending the whole file as 1 message? and is your producer code using sync or async mode? Cheers Rob. On 14 May 2014 15:49, Jun Rao jun...@gmail.com wrote: How many brokers and partitions do you have? You may try increasing batch.num.messages. Thanks, Jun On Tue, May 13, 2014 at 5:56 PM, Zhujie (zhujie, Smartcare) first.zhu...@huawei.com wrote: Dear all, We want to use kafka to collect and dispatch data file, but the performance is maybe lower than we want. In our cluster,there is a provider and a broker. We use a one thread read file from local disk of provider and send it to broker. The average throughput is only 3 MB/S~4MB/S. But if we just use java NIO API to send file ,the throughput can exceed 200MB/S. Why the kafka performance is so bad in our test, are we missing something?? Our server: Cpu: Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz*4 Mem:300G Disk:600G 15K RPM SAS*8 Configuration of provider: props.put(serializer.class, kafka.serializer.NullEncoder); props.put(metadata.broker.list, 169.10.35.57:9092); props.put(request.required.acks, 0); props.put(producer.type, async);//异步 props.put(queue.buffering.max.ms,500); props.put(queue.buffering.max.messages,10); props.put(batch.num.messages, 1200); props.put(queue.enqueue.timeout.ms, -1); props.put(send.buffer.bytes, 10240); Configuration of broker: # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the License); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # #http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an AS IS BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # see kafka.server.KafkaConfig for additional details and defaults # Server Basics # # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 # Socket Server Settings # # The port the socket server listens on port=9092 # Hostname the broker will bind to. If not set, the server will bind to all interfaces #host.name=localhost # Hostname the broker will advertise to producers and consumers. If not set, it uses the # value for host.name if configured. Otherwise, it will use the value returned from # java.net.InetAddress.getCanonicalHostName(). #advertised.host.name=hostname routable by clients # The port to publish to ZooKeeper for clients to use. If this is not set, # it will publish the same port that the broker binds to. #advertised.port=port accessible by clients # The number of threads handling network requests #num.network.threads=2 # The number of threads doing disk I/O #num.io.threads=8 # The send buffer (SO_SNDBUF) used by the socket server #socket.send.buffer.bytes=1048576 # The receive buffer (SO_RCVBUF) used by the socket server #socket.receive.buffer.bytes=1048576 # The maximum size of a request that the socket server will accept (protection against OOM) #socket.request.max.bytes=104857600 # Log Basics # # A comma seperated
Mbean kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec returning 0 always
Hi, We have a Kafka cluster which contains 3 kafka brokers. Data is pushed into kafka queue continously. To monitor the kafka brokers, we use the JMX metrics given by kafka. One thing i observed is there is one broker which is returning*0* all the time for the bean *kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec . * Is there some thing i should worry about? i am able to get the data from kafka and push the data into kafka with out a glitch. Thanks Arjun Narasimha Kota * *
Re: Mbean kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec returning 0 always
Hi Arjun ,
What is the configured number of logical partitions per topic per server .
With the partition size as 2 , Only two partitions would be created
distributed over 2 brokers with one partition each , you can confirm the
same by checking the log file directory for each broker (default -
/tmp/kafka-logs). Probably the 3d broker does not have any Topic-partition
for data to be written/read.
You may also use tool : bin/kafka-list-topic.sh –zookeeper
localhost:2181 –topic
{topic-name}
On Mon, May 26, 2014 at 4:02 PM, Arjun ar...@socialtwist.com wrote:
Hi,
We have a Kafka cluster which contains 3 kafka brokers. Data is pushed
into kafka queue continously.
To monitor the kafka brokers, we use the JMX metrics given by kafka.
One thing i observed is there is one broker which is returning*0* all
the time for the bean
*kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec
.
*
Is there some thing i should worry about? i am able to get the data from
kafka and push the data into kafka with out a glitch.
Thanks
Arjun Narasimha Kota
*
*
multi-broker topic rebalance
Hello! How can I rebalance topic within multiple brokers? For example: 1.I have about 30k consumers... 2.I need to use 3 broker (10k consumer per broker) 3. I use multiple (around 10) producers with default balance strategy Sometimes one broker have empty topic (consumer finished all message on broker)...But another broker have a lot of messages in topic... How to resolve such situation: -i cant just switch my consumers to another broker ( broker can not handle more than 10k consumers) -I can not find how to rebalance topic within multiple brokers... - I can not find how to setup limit for message per topic (or partition) ... If such things exists I can handle such case when add new message to broker in consumer side...and switch to another ... Thanks in advance, Aleksey Ryabkov
Re: Unable to scale kafka cluster
To add partitions, you need to use the --alter option in kafka-topics.sh.
Thanks,
Jun
On Sat, May 24, 2014 at 3:52 AM, Ertuğrul Yılmaz
ertugrul.yil...@convertale.com wrote:
Hi,
I know replica partition tools. I try to add new partition and assign to a
new broker during producer test run. but i can't.
*scripts are;*
bin/kafka-topics.sh --create --topic topic1 --partitions 2
--replication-factor 1 --zookeeper zookeeper:2184 zookeeper:2185
zookeeper:2186
cat topics-to-move.json
{topics:[{topic: topic1}],version:1}
bin/kafka-reassign-partitions.sh --topics-to-move-json-file
topics-to-move.json --broker-list 2 --generate --zookeeper zookeeper:2184
zookeeper:2185 zookeeper:2186
cat reassignment-json-file.json
{version:1,partitions:[{topic:topic1,partition:1,replicas:[2]}]}
bin/kafka-reassign-partitions.sh --reassignment-json-file
reassignment-json-file.json --execute --zookeeper zookeeper:2184
zookeeper:2185 zookeeper:2186
On Thu, May 22, 2014 at 5:34 PM, Jun Rao jun...@gmail.com wrote:
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIaddnewbrokersdynamicallytoacluster
?
Thanks,
Jun
On Thu, May 22, 2014 at 4:33 AM, Ertuğrul Yılmaz
ertugrul.yil...@convertale.com wrote:
Hi Kafka User and Dev team,
We want to use kafka to scale high traffic load. My plan is adding new
kafka instances to the cluster regarding the current load. In order to
achieve this, we have executed some producer tests and want to learn
how
we
can scale kafka cluster. Our test environments and configs are listed
below.
During our tests, nMsg.sec value did not increase after adding a new
broker. Even decreases.
Waiting for your comments.
Thanks.
*Topic descriptions are;*
Topic:tt3 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: tt3 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic:tt4 PartitionCount:2 ReplicationFactor:1 Configs:
Topic: tt4 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: tt4 Partition: 1 Leader: 2 Replicas: 2 Isr: 2
*Test code scripts are;*
bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
15 --topics tt3 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:52:04:523, 2014-05-22 08:52:30:668, 0, 1000, 200, 143.05,
5.4715, 15, 5737.2347
bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 15 --topics tt3 --threads 5 --message-size 1000
--batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:50:25:753, 2014-05-22 08:50:53:584, 0, 1000, 200, 143.05,
5.1400, 15, 5389.6734
bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
15 --topics tt4 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:48:18:628, 2014-05-22 08:48:47:153, 0, 1000, 200, 143.05,
5.0149, 15, 5258.5451
bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 15 --topics tt4 --threads 5 --message-size 1000
--batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:46:41:443, 2014-05-22 08:47:11:770, 0, 1000, 200, 143.05,
4.7170, 15, 4946.0876
bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
30 --topics tt3 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:42:03:119, 2014-05-22 08:42:45:885, 0, 1000, 200, 286.10,
6.6899, 30, 7014.9184
bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 30 --topics tt3 --threads 5 --message-size 1000
--batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:33:20:526, 2014-05-22 08:34:03:131, 0, 1000, 200, 286.10,
6.7152, 30, 7041.4271
bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
30 --topics tt4 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:43:41:984, 2014-05-22
Re: how many instances of the broker is suggested in one server.
The number of instances required will depend on the load. However, keep in mind that all instances on a server will compete for memory (e.g., for pagecache, jvm heap, etc), storage (log space) and CPU. Thanks, Jun On Sun, May 25, 2014 at 11:10 PM, Zhujie (zhujie, Smartcare) first.zhu...@huawei.com wrote: Hi all, I have one question, how many instances of the broker is suggested in one server? We start three instances, the performance will be three times faster than one instance. So how many instances should I start? Most appreciated zhujie
Re: multi-broker topic rebalance
Which version of Kafka are you using? How many partitions do you have in that topic? Thanks, Jun On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич a.ryab...@ntc-vulkan.ru wrote: Hello! How can I rebalance topic within multiple brokers? For example: 1.I have about 30k consumers... 2.I need to use 3 broker (10k consumer per broker) 3. I use multiple (around 10) producers with default balance strategy Sometimes one broker have empty topic (consumer finished all message on broker)...But another broker have a lot of messages in topic... How to resolve such situation: -i cant just switch my consumers to another broker ( broker can not handle more than 10k consumers) -I can not find how to rebalance topic within multiple brokers... - I can not find how to setup limit for message per topic (or partition) ... If such things exists I can handle such case when add new message to broker in consumer side...and switch to another ... Thanks in advance, Aleksey Ryabkov
Re: kafka broker failed to recovery from ZK failure
well. I just run the kafka-run-class.sh script with DeleteTopicCommand class directly. I noticed the warning on documentation http://DeleteTopicCommand page. but I thought it is only about not complete cleanup. I didn't know it can affect cluster health. maybe it's worthwhile to point it out the actual danger. On Thu, May 22, 2014 at 8:49 PM, Jun Rao jun...@gmail.com wrote: Delete topic is not supported in 0.8.1.1. How did you do it? Thanks, Jun On Thu, May 22, 2014 at 9:59 AM, Steven Wu steve...@netflix.com wrote: yes. I deleted a topic. but not at the time. it's a few hours before. On Fri, May 16, 2014 at 3:49 PM, Jun Rao jun...@gmail.com wrote: The problem is indicated by the following log in broker 1's controller log. Were you deleting any topic at that time? [2014-05-12 21:24:37,930] ERROR [BrokerChangeListener on Controller 1]: Error while handling broker changes (kafka.controller.ReplicaStateMachine$BrokerChangeListener) java.util.NoSuchElementException: key not found: [ocp.fillhealth.us-east-1,0] at scala.collection.MapLike$class.default(MapLike.scala:225) at scala.collection.mutable.HashMap.default(HashMap.scala:45) at scala.collection.MapLike$class.apply(MapLike.scala:135) at scala.collection.mutable.HashMap.apply(HashMap.scala:45) at kafka.controller.ControllerBrokerRequestBatch.updateMetadataRequestMapFor$1(ControllerChannelManager.scala:242) at kafka.controller.ControllerBrokerRequestBatch$$anonfun$addUpdateMetadataRequestForBrokers$1.apply(ControllerChannelManager.scala:268) at kafka.controller.ControllerBrokerRequestBatch$$anonfun$addUpdateMetadataRequestForBrokers$1.apply(ControllerChannelManager.scala:268) at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:99) at scala.collection.Iterator$class.foreach(Iterator.scala:772) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45) at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:99) at kafka.controller.ControllerBrokerRequestBatch.addUpdateMetadataRequestForBrokers(ControllerChannelManager.scala:268) at kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:940) at kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:386) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:342) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:330) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:330) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:329) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:328) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:328) at kafka.utils.Utils$.inLock(Utils.scala:538) at kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:327) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) Thanks, Jun On Tue, May 13, 2014 at 9:50 AM, Steven Wu steve...@netflix.com wrote: sorry for the wrong log file. please see the attached zip file for all 4 log files. On Mon, May 12, 2014 at 9:05 PM, Jun Rao jun...@gmail.com wrote: The controller log in broker 1 is too late. Could you send its log around 2014-05-12 21:24:37? Thanks, Jun On Mon, May 12, 2014 at 5:02 PM, Steven Wu steve...@netflix.com wrote: This is a three-node cluster. broker 0 lost connection to ZK. broker 1 does seem to take the controller role. but broker 0 stuck in the bad state and wasn't able to recover. it seems to start with these error msgs. I have attached complete controller and server log for broker 0 and 1. I am
HA: multi-broker topic rebalance
Hello! Which version of Kafka are you using? 1. I am using 0.8.1 How many partitions do you have in that topic? 2. In docs I found that number of partition must be equal or greater than number of consumers ... so I try 2 case: - 10k partition per broker - 20k partition per broker And one more question - how to force rebalance partitions to consumers in borker (and how much it cost...) I found in doc that consumer can Force itself to rebalance within in its consumer group but can not find API call...So for now I just reconnect With best regards, Aleksey Ryabkov -Исходное сообщение- От: Jun Rao [mailto:jun...@gmail.com] Отправлено: Tuesday, May 27, 2014 7:29 AM Кому: users@kafka.apache.org Тема: Re: multi-broker topic rebalance Which version of Kafka are you using? How many partitions do you have in that topic? Thanks, Jun On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич a.ryab...@ntc-vulkan.ru wrote: Hello! How can I rebalance topic within multiple brokers? For example: 1.I have about 30k consumers... 2.I need to use 3 broker (10k consumer per broker) 3. I use multiple (around 10) producers with default balance strategy Sometimes one broker have empty topic (consumer finished all message on broker)...But another broker have a lot of messages in topic... How to resolve such situation: -i cant just switch my consumers to another broker ( broker can not handle more than 10k consumers) -I can not find how to rebalance topic within multiple brokers... - I can not find how to setup limit for message per topic (or partition) ... If such things exists I can handle such case when add new message to broker in consumer side...and switch to another ... Thanks in advance, Aleksey Ryabkov
答复: how many instances of the broker is suggested in one server.
Is there any principles? -邮件原件- 发件人: Jun Rao [mailto:jun...@gmail.com] 发送时间: 2014年5月27日 11:25 收件人: users@kafka.apache.org 主题: Re: how many instances of the broker is suggested in one server. The number of instances required will depend on the load. However, keep in mind that all instances on a server will compete for memory (e.g., for pagecache, jvm heap, etc), storage (log space) and CPU. Thanks, Jun On Sun, May 25, 2014 at 11:10 PM, Zhujie (zhujie, Smartcare) first.zhu...@huawei.com wrote: Hi all, I have one question, how many instances of the broker is suggested in one server? We start three instances, the performance will be three times faster than one instance. So how many instances should I start? Most appreciated zhujie
