P.S. I guess the big question is what is the best way to handle or avoid
UNKNOWN_PRODUCER_ID when running versions that don’t include KAFKA-7190 /
KAFKA-8710 ?
We are using non-transactional idempotent Producers.
> On 9/03/2020, at 12:59 PM, James Olsen wrote:
>
> For completeness I have also
For completeness I have also tested 2.4.0 Broker with 2.4.0 Client. All works
correctly. Unfortunately as we are on AWS MSK we don’t have the option to use
2.4.0 for the Brokers.
So now I guess the question changes to what combo is best for us and will it
avoid UNKNOWN_PRODUCER_ID problems?
W
Jamie,
I’ve just tested with 2.3.1 Broker and 2.3.1 Client and it works correctly. So
with that setup it does deliver the batch as soon as any partition has data.
This is what we would expect from the Kafka docs.
So it looks like an issue with the 2.4.0 Client. This is concerning as I
wante
Hi James,
My understanding is that consumers will only ever have 1 in flight request to
each broker that has leader partitions of topics that it is subscribed to. The
fetch requests will ask for records for all leader partitions on the broker so
if the consumer is consuming from more than one pa
Using 2.3.1 Brokers makes things worse. There are now 2 fetch.max.wait.ms
delays before messages are delivered even though they were available at the
beginning.
2020-03-09 11:40:23,878 DEBUG
[org.apache.kafka.clients.consumer.internals.Fetcher]
'EE-ManagedThreadFactory-default-Thread-2' [Cons
Thanks for your response. Yes the second issue can be mitigated by reducing
the fetch.max.wait.ms although reducing it too far creates excessive CPU load
on the Brokers. However I've done some further testing and found what looks
like the underlying cause.
In the scenario below the Consumer i
Hi Eugen,
log.flush.interval.messages=1 will already give you the expected
behaviour - no need for the other two.
As a side note - log.flush.scheduler.interval.ms needs to be > 0.
Kafka accepts 0 as a value but Java's scheduled executor service
requires it to be > 0.
I am interested in your use
