Hi,
I have been playing around with different kafka configurations at producer,
broker and consumer side.
But could not figure out a way to get predictable low latency below 5 to 10
ms.
*Use Case:*
- Application has 100-200 producers, each producer is producing messages
at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec is
the max limit)
- Each producer is producing messages on unique topic on a same
partition so that ordering is maintained which is business requirement.
- Every producer has corresponding consumers. Some producers might have
1-10 consumers.
As I look at the above use case, I do not see I would be getting benefits
of batching which kafka is heavily depends on for getting maximum
throughput.
>From whole system I expect max throughput of around 40 - 50k but we would
like to achieve latency below 5-10 ms.
Is it possible with kafka to get this numbers of latency and throughput?
Note: I tried different configs focussing on latency part, but could not
get latency below 150ms, for example
kafka-clients {
linger.ms=0
acks=0
batch.size=1
}
akka.kafka.consumer {
poll-interval = 0
poll-timeout = 0
}
Thanks,
Pritam.
