[ https://issues.apache.org/jira/browse/KAFKA-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boyang Chen updated KAFKA-9605: ------------------------------- Description: In the Producer we could see network client hits fatal exception while trying to complete the batches after Txn manager hits fatal fenced error: {code:java} [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,673] ERROR [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Aborting producer batches due to fatal error (org.apache.kafka.clients.producer.internals.Sender) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,674] INFO [stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-0_0-producer, transactionalId=stream-soak-test-0_0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,684] INFO [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Resetting sequence number of batch with current sequence 354277 for partition windowed-node-counts-0 to 354276 (org.apache.kafka.clients.producer.internals.TransactionManager) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,684] INFO [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] Resetting sequence number of batch with current sequence 354277 for partition windowed-node-counts-0 to 354276 (org.apache.kafka.clients.producer.internals.ProducerBatch) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,685] ERROR [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Uncaught error in request completion: (org.apache.kafka.clients.NetworkClient) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) java.lang.IllegalStateException: Should not reopen a batch which is already aborted. at org.apache.kafka.common.record.MemoryRecordsBuilder.reopenAndRewriteProducerState(MemoryRecordsBuilder.java:295) at org.apache.kafka.clients.producer.internals.ProducerBatch.resetProducerState(ProducerBatch.java:395) at org.apache.kafka.clients.producer.internals.TransactionManager.lambda$adjustSequencesDueToFailedBatch$4(TransactionManager.java:770) at org.apache.kafka.clients.producer.internals.TransactionManager$TopicPartitionEntry.resetSequenceNumbers(TransactionManager.java:180) at org.apache.kafka.clients.producer.internals.TransactionManager.adjustSequencesDueToFailedBatch(TransactionManager.java:760) at org.apache.kafka.clients.producer.internals.TransactionManager.handleFailedBatch(TransactionManager.java:735) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:671) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:662) at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:620) at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:554) at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:69) at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:745) at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:571) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:563) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:304) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at java.lang.Thread.run(Thread.java:748) {code} The proper fix is to add a check for handle failed batch in txn manager. was: ``` [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,673] ERROR [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Aborting producer batches due to fatal error (org.apache.kafka.clients.producer.internals.Sender) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,674] INFO [stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-0_0-producer, transactionalId=stream-soak-test-0_0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,684] INFO [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Resetting sequence number of batch with current sequence 354277 for partition windowed-node-counts-0 to 354276 (org.apache.kafka.clients.producer.internals.TransactionManager) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,684] INFO [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] Resetting sequence number of batch with current sequence 354277 for partition windowed-node-counts-0 to 354276 (org.apache.kafka.clients.producer.internals.ProducerBatch) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 21:23:28,685] ERROR [kafka-producer-network-thread | stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] [Producer clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Uncaught error in request completion: (org.apache.kafka.clients.NetworkClient) [2020-02-24T13:23:29-08:00] (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) java.lang.IllegalStateException: Should not reopen a batch which is already aborted. at org.apache.kafka.common.record.MemoryRecordsBuilder.reopenAndRewriteProducerState(MemoryRecordsBuilder.java:295) at org.apache.kafka.clients.producer.internals.ProducerBatch.resetProducerState(ProducerBatch.java:395) at org.apache.kafka.clients.producer.internals.TransactionManager.lambda$adjustSequencesDueToFailedBatch$4(TransactionManager.java:770) at org.apache.kafka.clients.producer.internals.TransactionManager$TopicPartitionEntry.resetSequenceNumbers(TransactionManager.java:180) at org.apache.kafka.clients.producer.internals.TransactionManager.adjustSequencesDueToFailedBatch(TransactionManager.java:760) at org.apache.kafka.clients.producer.internals.TransactionManager.handleFailedBatch(TransactionManager.java:735) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:671) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:662) at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:620) at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:554) at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:69) at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:745) at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:571) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:563) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:304) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at java.lang.Thread.run(Thread.java:748)``` > EOS Producer could throw illegal state if trying to complete a failed batch > after fatal error > --------------------------------------------------------------------------------------------- > > Key: KAFKA-9605 > URL: https://issues.apache.org/jira/browse/KAFKA-9605 > Project: Kafka > Issue Type: Bug > Affects Versions: 2.4.0, 2.5.0 > Reporter: Boyang Chen > Assignee: Boyang Chen > Priority: Major > Fix For: 2.5.0 > > > In the Producer we could see network client hits fatal exception while trying > to complete the batches after Txn manager hits fatal fenced error: > {code:java} > > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 > 21:23:28,673] ERROR [kafka-producer-network-thread | > stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] > [Producer > clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, > transactionalId=stream-soak-test-1_0] Aborting producer batches due to fatal > error (org.apache.kafka.clients.producer.internals.Sender) > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) > org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an > operation with an old epoch. Either there is a newer producer with the same > transactionalId, or the producer's transaction has been expired by the broker. > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 > 21:23:28,674] INFO > [stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3] > [Producer > clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-0_0-producer, > transactionalId=stream-soak-test-0_0] Closing the Kafka producer with > timeoutMillis = 9223372036854775807 ms. > (org.apache.kafka.clients.producer.KafkaProducer) > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 > 21:23:28,684] INFO [kafka-producer-network-thread | > stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] > [Producer > clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, > transactionalId=stream-soak-test-1_0] Resetting sequence number of batch > with current sequence 354277 for partition windowed-node-counts-0 to 354276 > (org.apache.kafka.clients.producer.internals.TransactionManager) > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 > 21:23:28,684] INFO [kafka-producer-network-thread | > stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] > Resetting sequence number of batch with current sequence 354277 for > partition windowed-node-counts-0 to 354276 > (org.apache.kafka.clients.producer.internals.ProducerBatch) > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 > 21:23:28,685] ERROR [kafka-producer-network-thread | > stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer] > [Producer > clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer, > transactionalId=stream-soak-test-1_0] Uncaught error in request completion: > (org.apache.kafka.clients.NetworkClient) > [2020-02-24T13:23:29-08:00] > (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) > java.lang.IllegalStateException: Should not reopen a batch which is already > aborted. > at > org.apache.kafka.common.record.MemoryRecordsBuilder.reopenAndRewriteProducerState(MemoryRecordsBuilder.java:295) > at > org.apache.kafka.clients.producer.internals.ProducerBatch.resetProducerState(ProducerBatch.java:395) > at > org.apache.kafka.clients.producer.internals.TransactionManager.lambda$adjustSequencesDueToFailedBatch$4(TransactionManager.java:770) > at > org.apache.kafka.clients.producer.internals.TransactionManager$TopicPartitionEntry.resetSequenceNumbers(TransactionManager.java:180) > at > org.apache.kafka.clients.producer.internals.TransactionManager.adjustSequencesDueToFailedBatch(TransactionManager.java:760) > at > org.apache.kafka.clients.producer.internals.TransactionManager.handleFailedBatch(TransactionManager.java:735) > at > org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:671) > at > org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:662) > at > org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:620) > at > org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:554) > at > org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:69) > at > org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:745) > at > org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) > at > org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:571) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:563) > at > org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:304) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) > at java.lang.Thread.run(Thread.java:748) > {code} > The proper fix is to add a check for handle failed batch in txn manager. -- This message was sent by Atlassian Jira (v8.3.4#803005)