[ https://issues.apache.org/jira/browse/KAFKA-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816800#comment-17816800 ]
Greg Harris commented on KAFKA-13338: ------------------------------------- Apache Kafka does not have 5.x or 6.x releases, so you must be using a fork of Kafka. The log message and your remediation are as-expected; Connect must use compacted internal topics, and enforcement of this was added after-the-fact. The delay between the upgrade and the connectors failing also indicates that the two are unrelated, and compaction shouldn't be able to cause a InvalidRecordException. As this issue is so stale, and there isn't any information to proceed on, I have closed it. Feel free to raise a new issue if this problem reappears. > kafka upgrade 6.1.1 asking to change log.cleanup.policy from delete to compact > ------------------------------------------------------------------------------ > > Key: KAFKA-13338 > URL: https://issues.apache.org/jira/browse/KAFKA-13338 > Project: Kafka > Issue Type: Bug > Components: log > Reporter: Fátima Galera > Priority: Major > Attachments: image003.png > > > Hi all, > > During kafka upgrade from 5.3.1 to 6.1.1.1 we get below error starting > kafka-connector services > > ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception > in herder work thread, exiting: > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > 87738-org.apache.kafka.common.config.ConfigException: Topic 'connect-offsets' > supplied via the 'offset.storage.topic' property is required to have > 'cleanup.policy=compact' to guarantee consistency and durability of source > connector offsets, but found the topic currently has 'cleanup.policy=delete'. > Continuing would likely result in eventually losing source connector offsets > and problems restarting this Connect cluster in the future. Change the > 'offset.storage.topic' property in the Connect worker configurations to use a > topic with 'cleanup.policy=compact'. > > After that error we added log.cleanup.policy=compact parameter on > /etc/kafka/server.properties and kafka-connector service started but a few > days later all the connectors are down with attached error. > > Could you please let us know how we can resolved the issue? > > Best regards and thanks > Fátima > NGA -Alight > > -- This message was sent by Atlassian Jira (v8.20.10#820010)