[jira] [Commented] (KAFKA-15931) Cached transaction index gets closed if tiered storage read is interrupted
[ https://issues.apache.org/jira/browse/KAFKA-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809148#comment-17809148 ] Jorge Esteban Quilcate Otoya commented on KAFKA-15931: -- [~satish.duggana] [~ckamal] I've opened a new PR to deal with this – have a look: https://github.com/apache/kafka/pull/15241 > Cached transaction index gets closed if tiered storage read is interrupted > -- > > Key: KAFKA-15931 > URL: https://issues.apache.org/jira/browse/KAFKA-15931 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Ivan Yurchenko >Assignee: Jorge Esteban Quilcate Otoya >Priority: Minor > > This reproduces when reading from remote storage with the default > {{fetch.max.wait.ms}} (500) or lower. {{isolation.level=read_committed}} is > needed to trigger this. > It's not easy to reproduce on local-only setups, unfortunately, because reads > are fast and aren't interrupted. > This error is logged > {noformat} > [2023-11-29 14:01:01,166] ERROR Error occurred while reading the remote data > for topic1-0 (kafka.log.remote.RemoteLogReader) > org.apache.kafka.common.KafkaException: Failed read position from the > transaction index > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:235) > at > org.apache.kafka.storage.internals.log.TransactionIndex.collectAbortedTxns(TransactionIndex.java:171) > at > kafka.log.remote.RemoteLogManager.collectAbortedTransactions(RemoteLogManager.java:1359) > at > kafka.log.remote.RemoteLogManager.addAbortedTransactions(RemoteLogManager.java:1341) > at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1310) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.nio.channels.ClosedChannelException > at > java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:325) > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:233) > ... 10 more > {noformat} > and after that this txn index becomes unusable until the process is restarted. > I suspect, it's caused by the reading thread being interrupted due to the > fetch timeout. At least [this > code|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L159-L160] > in {{AbstractInterruptibleChannel}} is called. > Fixing may be easy: reopen the channel in {{TransactionIndex}} if it's close. > However, off the top of my head I can't say if there are some less obvious > implications of this change. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15931) Cached transaction index gets closed if tiered storage read is interrupted
[ https://issues.apache.org/jira/browse/KAFKA-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17791514#comment-17791514 ] Satish Duggana commented on KAFKA-15931: Sure [~ivanyu] , added it to https://issues.apache.org/jira/browse/KAFKA-15420 > Cached transaction index gets closed if tiered storage read is interrupted > -- > > Key: KAFKA-15931 > URL: https://issues.apache.org/jira/browse/KAFKA-15931 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Ivan Yurchenko >Priority: Minor > > This reproduces when reading from remote storage with the default > {{fetch.max.wait.ms}} (500) or lower. {{isolation.level=read_committed}} is > needed to trigger this. > It's not easy to reproduce on local-only setups, unfortunately, because reads > are fast and aren't interrupted. > This error is logged > {noformat} > [2023-11-29 14:01:01,166] ERROR Error occurred while reading the remote data > for topic1-0 (kafka.log.remote.RemoteLogReader) > org.apache.kafka.common.KafkaException: Failed read position from the > transaction index > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:235) > at > org.apache.kafka.storage.internals.log.TransactionIndex.collectAbortedTxns(TransactionIndex.java:171) > at > kafka.log.remote.RemoteLogManager.collectAbortedTransactions(RemoteLogManager.java:1359) > at > kafka.log.remote.RemoteLogManager.addAbortedTransactions(RemoteLogManager.java:1341) > at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1310) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.nio.channels.ClosedChannelException > at > java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:325) > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:233) > ... 10 more > {noformat} > and after that this txn index becomes unusable until the process is restarted. > I suspect, it's caused by the reading thread being interrupted due to the > fetch timeout. At least [this > code|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L159-L160] > in {{AbstractInterruptibleChannel}} is called. > Fixing may be easy: reopen the channel in {{TransactionIndex}} if it's close. > However, off the top of my head I can't say if there are some less obvious > implications of this change. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15931) Cached transaction index gets closed if tiered storage read is interrupted
[ https://issues.apache.org/jira/browse/KAFKA-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17791140#comment-17791140 ] Kamal Chandraprakash commented on KAFKA-15931: -- [~ivanyu] Thanks for filing this issue! While working on KAFKA-15047, we noticed that some of the [TransactionsWithTieredStoreTest|https://ge.apache.org/s/ofqqovlfxqpwa/tests/task/:storage:test/details/org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest] failed with this error. > Cached transaction index gets closed if tiered storage read is interrupted > -- > > Key: KAFKA-15931 > URL: https://issues.apache.org/jira/browse/KAFKA-15931 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Ivan Yurchenko >Priority: Minor > > This reproduces when reading from remote storage with the default > {{fetch.max.wait.ms}} (500) or lower. {{isolation.level=read_committed}} is > needed to trigger this. > It's not easy to reproduce on local-only setups, unfortunately, because reads > are fast and aren't interrupted. > This error is logged > {noformat} > [2023-11-29 14:01:01,166] ERROR Error occurred while reading the remote data > for topic1-0 (kafka.log.remote.RemoteLogReader) > org.apache.kafka.common.KafkaException: Failed read position from the > transaction index > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:235) > at > org.apache.kafka.storage.internals.log.TransactionIndex.collectAbortedTxns(TransactionIndex.java:171) > at > kafka.log.remote.RemoteLogManager.collectAbortedTransactions(RemoteLogManager.java:1359) > at > kafka.log.remote.RemoteLogManager.addAbortedTransactions(RemoteLogManager.java:1341) > at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1310) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.nio.channels.ClosedChannelException > at > java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:325) > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:233) > ... 10 more > {noformat} > and after that this txn index becomes unusable until the process is restarted. > I suspect, it's caused by the reading thread being interrupted due to the > fetch timeout. At least [this > code|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L159-L160] > in {{AbstractInterruptibleChannel}} is called. > Fixing may be easy: reopen the channel in {{TransactionIndex}} if it's close. > However, off the top of my head I can't say if there are some less obvious > implications of this change. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15931) Cached transaction index gets closed if tiered storage read is interrupted
[ https://issues.apache.org/jira/browse/KAFKA-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17791067#comment-17791067 ] Ivan Yurchenko commented on KAFKA-15931: [~satishd] how do you feel: should this be included in https://issues.apache.org/jira/browse/KAFKA-15420 ? > Cached transaction index gets closed if tiered storage read is interrupted > -- > > Key: KAFKA-15931 > URL: https://issues.apache.org/jira/browse/KAFKA-15931 > Project: Kafka > Issue Type: Bug > Components: Tiered-Storage >Affects Versions: 3.6.0 >Reporter: Ivan Yurchenko >Priority: Minor > > This reproduces when reading from remote storage with the default > {{fetch.max.wait.ms}} (500) or lower. {{isolation.level=read_committed}} is > needed to trigger this. > It's not easy to reproduce on local-only setups, unfortunately, because reads > are fast and aren't interrupted. > This error is logged > {noformat} > [2023-11-29 14:01:01,166] ERROR Error occurred while reading the remote data > for topic1-0 (kafka.log.remote.RemoteLogReader) > org.apache.kafka.common.KafkaException: Failed read position from the > transaction index > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:235) > at > org.apache.kafka.storage.internals.log.TransactionIndex.collectAbortedTxns(TransactionIndex.java:171) > at > kafka.log.remote.RemoteLogManager.collectAbortedTransactions(RemoteLogManager.java:1359) > at > kafka.log.remote.RemoteLogManager.addAbortedTransactions(RemoteLogManager.java:1341) > at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1310) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62) > at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.nio.channels.ClosedChannelException > at > java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:325) > at > org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:233) > ... 10 more > {noformat} > and after that this txn index becomes unusable until the process is restarted. > I suspect, it's caused by the reading thread being interrupted due to the > fetch timeout. At least [this > code|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L159-L160] > in {{AbstractInterruptibleChannel}} is called. > Fixing may be easy: reopen the channel in {{TransactionIndex}} if it's close. > However, off the top of my head I can't say if there are some less obvious > implications of this change. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)