Michał Łukowicz created KAFKA-12274:
---------------------------------------
Summary: Transactional operation fails when broker is replaced
using the same broker ID.
Key: KAFKA-12274
URL: https://issues.apache.org/jira/browse/KAFKA-12274
Project: Kafka
Issue Type: Bug
Components: controller, producer
Affects Versions: 2.6.0
Reporter: Michał Łukowicz
Hello Team!
One of our clusters is being used to:
* process transactional writes
* had ack set to all
We are using java client and followed all recommendation regarding avoiding
dead fencing issues, etc.
We spotted the problem during upgrading kafka hosts to stronger machines:
* stop old broker
*
start a new clean broker node (a different hostname) reusing the same broker.id
During the operation we found that while kafka is normally replicating
partitions to recover after very short period of time (1 - 3 mins) we start to
see error on kafka broker:
broker=13] Error processing append operation on partition <partition>
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order
sequence number for producerId 51119 at offset 16878080903 in partition
<partition>: 2930356 (incoming seq. number), 2930213 (current end sequence
number)
And we are starting to see records buffered on the Producer side, and
eventually, the producer send requests failed with::
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 2
record(s) for <topic>:120892 ms has passed since batch creation
The only additional thing we observed is that for some reason couple of
paritions ISR had been reduced to 1.
The same situation can be observed when adding new brokers to cluster and
performing rebalacing (using kafka cruise control) and setting concurrent
partition and leader movements to higher value.
Can you please let me know if this is a bug ... or we are doing something wrong?
Kafka 2.6.0
min.insync.replica for topics is set to 1
replication.factor is 3
all transaction settings are currently default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
