Re: Kafka null keys - OK or a problem?

2016-10-10 Thread Michael Noll
Depends on which partitioner you are using, see [1] and [2]. From what I understand the `NewHashPartitioner` comes closest to the behavior of Kafka Java producer, but instead of going round-robin for null-keyed messages it picks a partition at random. [1] https://godoc.org/github.com/Shopify/sa

Re: Kafka null keys - OK or a problem?

2016-10-09 Thread Ali Akhtar
It seems to be using a Hash Partitioner here: https://github.com/Shopify/sarama/blob/master/config.go#L262 and HashPartitioner is documented as: > If the message's key is nil then a random partition is chosen https://godoc.org/github.com/Shopify/sarama#example-Partitioner--Random So.. it shoul

Re: Kafka null keys - OK or a problem?

2016-10-09 Thread Ali Akhtar
Hey Michael, We're using this one: https://github.com/Shopify/sarama Any ideas how that one works? On Mon, Oct 10, 2016 at 11:48 AM, Michael Noll wrote: > FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as > follows: If no partition is explicitly specified (to send the

Re: Kafka null keys - OK or a problem?

2016-10-09 Thread Michael Noll
FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as follows: If no partition is explicitly specified (to send the message to) AND the key is null, then the DefaultPartitioner [1] will assign messages to topic partitions in a round-robin fashion. See the javadoc and also the

Re: Kafka null keys - OK or a problem?

2016-10-09 Thread Ali Akhtar
If keys are null, what happens in terms of partitioning, is the load spread evenly..? On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira wrote: > Kafka itself supports null keys. I'm not sure about the Go client you > use, but Confluent's Go client also supports null keys > (https://github.com/conflu

Re: Kafka null keys - OK or a problem?

2016-10-09 Thread Gwen Shapira
Kafka itself supports null keys. I'm not sure about the Go client you use, but Confluent's Go client also supports null keys (https://github.com/confluentinc/confluent-kafka-go/). If you decide to generate keys and you want even spread, a random number generator is probably your best bet. Gwen O

Kafka null keys - OK or a problem?

2016-10-09 Thread Ali Akhtar
A kafka producer written elsewhere that I'm using, which uses the Go kafka driver, is sending messages where the key is null. Is this OK - or will this cause issues due to partitioning not happening correctly? What would be a good way to generate keys in this case, to ensure even partition spread