[ https://issues.apache.org/jira/browse/KAFKA-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kamal Chandraprakash updated KAFKA-16780: ----------------------------------------- Summary: Txn consumer exerts pressure on remote storage when collecting aborted transaction (was: Txn consumer exerts pressure on remote storage when reading non-txn topic) > Txn consumer exerts pressure on remote storage when collecting aborted > transaction > ---------------------------------------------------------------------------------- > > Key: KAFKA-16780 > URL: https://issues.apache.org/jira/browse/KAFKA-16780 > Project: Kafka > Issue Type: Task > Reporter: Kamal Chandraprakash > Assignee: Kamal Chandraprakash > Priority: Major > Labels: kip > > h3. Logic to read aborted txns: > # When the consumer enables isolation_level as {{READ_COMMITTED}} and reads > a non-txn topic, then the broker has to > [traverse|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LocalLog.scala#L394] > all the local log segments to collect the aborted transactions since there > won't be any entry in the transaction index. > # The same > [logic|https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1436] > is applied while reading from remote storage. In this case, when the FETCH > request is reading data from the first remote log segment, then it has to > fetch the transaction indexes of all the remaining remote-log segments, and > then the call lands to the local-log segments before responding to the FETCH > request which increases the time taken to serve the requests. > The [EoS Abort > Index|https://docs.google.com/document/d/1Rlqizmk7QCDe8qAnVW5e5X8rGvn6m2DCR3JR2yqwVjc] > design doc explains how the transaction index file filters out the aborted > transaction records. > The issue is when consumers are enabled with the {{READ_COMMITTED}} isolation > level but read the non-txn topics. If the topic is enabled with the > transaction, then we expect the transaction to either commit/rollback within > 15 minutes (default transaction.max.timeout.ms = 15 mins), possibly we may > have to search only a few remote log segments to collect the aborted txns. -- This message was sent by Atlassian Jira (v8.20.10#820010)