[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039623#comment-17039623
]
Boquan Tang commented on KAFKA-9142:
Is this the similar to https://issues.apache.org/jira/browse/KAFKA-8335 ? The
issue is resolved after 2.2.1 so you may want to update your broker.
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[ https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019176#comment-17019176 ] panpan.liu commented on KAFKA-9142: --- [~guozhang] Yes. Before I restart broker, I found there is no cleaner thread by jstack. This is loading log after restart broker. ``` /server.log.2020-01-15-08:[2020-01-15 08:32:26,819] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 223641 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,017] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 223839 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,140] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 223962 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,281] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 224103 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,399] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 224221 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,509] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 224331 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,693] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 224515 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,863] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 224685 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:27,993] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 224815 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:28,313] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 225135 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:28,456] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 225278 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:28,656] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 225478 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:28,982] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 225804 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:29,167] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 225989 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:29,493] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 226315 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:29,506] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 226328 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:29,715] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 226537 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:29,919] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 226741 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:30,149] INFO [Transaction State Manager 0]: Finished loading 9 transaction metadata from __transaction_state-24 in 226971 milliseconds (kafka.coordinator.transaction.TransactionStateManager) ./server.log.2020-01-15-08:[2020-01-15 08:32:30,424] INFO [Transaction State Manager 0]: Finished
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019170#comment-17019170
]
Guozhang Wang commented on KAFKA-9142:
--
Thanks [~panpan.liu], by `recover to normal` did you mean that after restarting
the deletion thread seems to work the log size does gets reduced?
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018849#comment-17018849
]
panpan.liu commented on KAFKA-9142:
---
I restart brokers one by one. And it recovery to normal.
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968285#comment-16968285
]
panpan.liu commented on KAFKA-9142:
---
[~guozhang] [~boquan] could please see the issue
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967944#comment-16967944
]
Francesca Capochiani commented on KAFKA-9142:
-
Same issue too!
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[ https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967312#comment-16967312 ] panpan.liu commented on KAFKA-9142: --- I have met the same issue: ["__consumer_offsets replicas grow way too big"|https://issues.apache.org/jira/browse/KAFKA-3917] > topic __transaction_state disk space grows too big > -- > > Key: KAFKA-9142 > URL: https://issues.apache.org/jira/browse/KAFKA-9142 > Project: Kafka > Issue Type: Improvement > Components: log cleaner >Affects Versions: 2.0.0 >Reporter: panpan.liu >Priority: Critical > Attachments: image-2019-11-05-15-31-46-859.png, > image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png > > > kafka broker :2.0.0 > kafka stream client: 2.1.0 > topic description please see attachements. > The broker runs from 10/2018. I find the topic grows up and does not delete > (*cleanup.policy=compact*). > How can I handle it ? > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
