[ https://issues.apache.org/jira/browse/KAFKA-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias J. Sax updated KAFKA-16221: ------------------------------------ Description: https://issues.apache.org/jira/browse/KAFKA-14831 fixed a producer bug about internal TX state transition and the producer is now throwing an IllegalStateException in situations it did swallow an internal error before. This change surfaces a bug in Kafka Streams: Kafka Streams calls `abortTransaction()` blindly when a task is closed dirty, even if the Producer is already in an internal fatal state. However, if the Producer is in a fatal state, Kafka Streams should skip `abortTransaction` and only `close()` the Producer when closing a task dirty. The bug is surfaced after `commitTransaction()` did timeout and another error happens leading to the call to `abortTransaction()` – Kafka Streams does not track right now if a commit-TX is in progress. If the Producer throws an IllegalStateException on `abortTransaction()` Kafka Streams treats this exception ("correctly") as fatal, and StreamsThread dies. However, Kafka Streams is actually in a state in which it can recover from, and thus should not let StreamThread die by carry forward (by not calling `abortTransaction()` and moving forward with the dirty close of the task). was: https://issues.apache.org/jira/browse/KAFKA-14831 fixed a producer bug about internal TX state transition and the producer is now throwing an IllegalStateException in situations it did swallow an internal error before. This change surfaces a bug in Kafka Streams: Kafka Streams calls `abortTransaction()` blindly when a task is closed dirty, even if the Producer is already in an internal fatal state. However, if the Producer is in a fatal state, Kafka Streams should skip `abortTransaction` and only `close()` the Producer when closing a task dirty. If the Producer throws an IllegalStateException on `abortTransaction()` Kafka Streams treats this exception ("correctly") as fatal, and StreamsThread dies. However, Kafka Streams is actually in a state in which it can recover from, and thus should not let StreamThread die by carry forward (by not calling `abortTransaction()` and moving forward with the dirty close of the task). > IllegalStateException from Producer > ----------------------------------- > > Key: KAFKA-16221 > URL: https://issues.apache.org/jira/browse/KAFKA-16221 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 3.6.0 > Reporter: Matthias J. Sax > Priority: Critical > > https://issues.apache.org/jira/browse/KAFKA-14831 fixed a producer bug about > internal TX state transition and the producer is now throwing an > IllegalStateException in situations it did swallow an internal error before. > This change surfaces a bug in Kafka Streams: Kafka Streams calls > `abortTransaction()` blindly when a task is closed dirty, even if the > Producer is already in an internal fatal state. However, if the Producer is > in a fatal state, Kafka Streams should skip `abortTransaction` and only > `close()` the Producer when closing a task dirty. > The bug is surfaced after `commitTransaction()` did timeout and another error > happens leading to the call to `abortTransaction()` – Kafka Streams does not > track right now if a commit-TX is in progress. > If the Producer throws an IllegalStateException on `abortTransaction()` Kafka > Streams treats this exception ("correctly") as fatal, and StreamsThread dies. > However, Kafka Streams is actually in a state in which it can recover from, > and thus should not let StreamThread die by carry forward (by not calling > `abortTransaction()` and moving forward with the dirty close of the task). -- This message was sent by Atlassian Jira (v8.20.10#820010)