Kafka uses murmur2 key hashing by default. You can also create your own custom partitioner. The partitioner can be set on per producer / consumer basis. -- Best regards, Rad
On Sat, Sep 17, 2016 at 11:01 AM +0200, "kant kodali" <kanth...@gmail.com> wrote: so Zookeeper will select which broker it should direct the message to if I have 3 brokers and 3 partitions of a topic? I only finished the benchmarks of Kafka and NSQ and still working NATS Streaming Server (one more day I will finish it). But so far Kafka had a great throughput with Java Client about 55000 messages per second(receive throughput) and 70K messages per second(sendthrougput)each message size is 1KB. so yeah something wrong with node.js client NSQ was ~ 3K receive throughput and~4K send throughput one thing about my NSQ benchmark is first of all I am new to Go and I have the following code (pasting only the lines that are relevant). I just dont know if w.Publish is a blocking call? If it is a blocking call I have to see if the library offers any non-blocking async call since the java version I wrote for Kafka was async. That way I am comparing apples to apples w, _ := nsq.NewProducer("172.31.18.175:4150", config) for i := 0; i<500000; i++ { err := w.Publish("nsq_go_test", []byte(data)) if err != nil { log.Panic("Could not connect") } } >From what I heard beating NATS is going to be harder but I rather run my own experiment than going with what I heard. Also it On Sat, Sep 17, 2016 1:38 AM, Ali Akhtar ali.rac...@gmail.com wrote: You can create multiple partitions of a topic and kafka will attempt to distribute them evenly. E.g if you have 3 brokers and you create 3 partitions of a topic, each broker will be the leader of 1 of the 3 partitions. P.S how did the benchmarking go? On Sat, Sep 17, 2016 at 1:36 PM, kant kodali wrote: > can one topic be registered in multiple brokers? if so, which component of > kafka decides which broker should get the message for that particular > topic? > Thanks!