[jira] [Updated] (KAFKA-14055) Transaction markers may be lost during cleaning if data keys conflict with marker keys
[ https://issues.apache.org/jira/browse/KAFKA-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-14055: Affects Version/s: 2.8.1 2.7.2 2.6.3 2.5.1 2.4.1 > Transaction markers may be lost during cleaning if data keys conflict with > marker keys > -- > > Key: KAFKA-14055 > URL: https://issues.apache.org/jira/browse/KAFKA-14055 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.4.1, 2.5.1, 2.6.3, 2.7.2, 2.8.1, 3.0.1, 3.2.0, 3.1.1 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Critical > Fix For: 3.3.0, 3.0.2, 3.1.2, 3.2.1 > > > We have been seeing recently hanging transactions occur on streams changelog > topics quite frequently. After investigation, we found that the keys used in > the changelog topic conflict with the keys used in the transaction markers > (the schema used in control records is 4 bytes, which happens to be the same > for the changelog topics that we investigated). When we build the offset map > prior to cleaning, we do properly exclude the transaction marker keys, but > the bug is the fact that we do not exclude them during the cleaning phase. > This can result in the marker being removed from the cleaned log before the > corresponding data is removed when there is a user record with a conflicting > key at a higher offset. A side effect of this is a so-called "hanging" > transaction, but the bigger problem is that we lose the atomicity of the > transaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14055) Transaction markers may be lost during cleaning if data keys conflict with marker keys
[ https://issues.apache.org/jira/browse/KAFKA-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-14055: Affects Version/s: 3.1.1 3.2.0 3.0.1 > Transaction markers may be lost during cleaning if data keys conflict with > marker keys > -- > > Key: KAFKA-14055 > URL: https://issues.apache.org/jira/browse/KAFKA-14055 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.0.1, 3.2.0, 3.1.1 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Critical > Fix For: 3.3.0, 3.0.2, 3.1.2, 3.2.1 > > > We have been seeing recently hanging transactions occur on streams changelog > topics quite frequently. After investigation, we found that the keys used in > the changelog topic conflict with the keys used in the transaction markers > (the schema used in control records is 4 bytes, which happens to be the same > for the changelog topics that we investigated). When we build the offset map > prior to cleaning, we do properly exclude the transaction marker keys, but > the bug is the fact that we do not exclude them during the cleaning phase. > This can result in the marker being removed from the cleaned log before the > corresponding data is removed when there is a user record with a conflicting > key at a higher offset. A side effect of this is a so-called "hanging" > transaction, but the bigger problem is that we lose the atomicity of the > transaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
