I am not a huge fan of criticizing without making myself useful first;
so here is what you can do in order to have a producer writing records
to a specific broker:
1. Create a topic with the # of partitions equal to the # of brokers. By
default Kafka will try to evenly distributed the partitions across the
brokers so it is expected that each broker will host at least one
partition of the topic.
2. Figure out which broker host the leader of the partition. Each
partition will always have one leader and multiple replicas. The leader
(where all the writes goes by) will live in one of the brokers. You can
figure this out by running the `kafka-topics` command using the option
`--describe`. Your job here is figuring the partition id and not the
broker id.
3. In the producer code, send records using one of the following approach:
ProducerRecord<K, V> record = new ProducerRecord(topic,
*partitionId*, K, V);
producer.send(record);
4. There might be situations in which the leader of a given partition
may be moved to another broker, in which you will have to keep track of
this movement and change your code accordingly. You can reasonably
achieve a dynamic behavior in your producer code by fetching the
partitions using Kafka's admin API before executing the send() command
to check if things have changed.
As you probably learn at this point, in Kafka the unit of writing is not
the broker but a partition that in turn lives in one of the brokers.
Not here comes the critic:
- *Don't Do This!*
You will be giving up of all the logic that Kafka puts in place to
ensure horizontal scalability and fault-tolerance. This practice might
be common in some messaging technologies but it isn't in Kafka. Not to
mention that you would have to have an considerable Ops effort to keep
track of partitions in brokers, as well as write a more sophisticated
code to protect you against situations that would easily break all this
logic such as growing/shrinking the # of brokers and/or partitions.
Likely, Kafka is not the appropriate technology for you in this case.
Thanks,
-- Ricardo
On 7/26/20 11:11 AM, Rajib Deb wrote:
Hi,
I came across the below question and wanted to seek an answer on the same.
If a producer needs to write to a certain broker only, is this possible. For
example, if the producer is in Europe, it will write to the broker near to
Europe, if US it will write to broker near to US. But consumers should be able
to read from both the topics.
Thanks
Rajib
