One more reply to myself.
For reasons that are not quite clear we found that there were log
directories left on brokers that corresponded to topics or partitions that
no longer exist or that no longer reside on a given broker.
Those directories don't and won't have a topicID which is causing kafka to
fail when running as a broker in k-raft mode.
I created this script:
> #!/bin/bashset -e -u -o pipefail
> unset KAFKA_OPTS; unset JMX_PORT
>
> cd /opt/kafka
>
> rm -f /tmp/topics_and_parts
>
> ./bin/kafka-topics.sh --describe --bootstrap-server broker:9092 | grep
> 'Topic:' |grep 'Partition:' >/tmp/topics_and_parts
>
> id=${HOSTNAME##kafka-}
> pushd data/topics &>/dev/null
> for logDir in *; do
> topic=${logDir%-*}
> partition=${logDir##*-}
> if [ ! -d "$logDir" ]; then
> continue
> fi
> if [ "$topic" == "__cluster_metadata" ]; then
> continue
> fi
> if [ "$partition" == "delete" ]; then
> continue
> fi
> if [ "$partition" == "stray" ]; then
> continue
> fi
> if ! grep -q -E "Topic: ${topic}[ ]*Partition: ${partition}"
> /tmp/topics_and_parts; then
> echo "$logDir: topic or partition does not exist"
> else
> # the topic and partition exists, check if it should be on this broker
> line=$(grep -E "Topic: ${topic}[ ]*Partition: ${partition}"
> /tmp/topics_and_parts)
> isr=$(echo "$line" |awk '{print $10}')
> found=0
> for i in $(echo "$isr" |tr "," " "); do
> if [ "$i" == "$id" ]; then
> found=1
> break
> fi
> done
> if [ "$found" == "0" ]; then
> echo "$logDir: should not be on this broker"
> fi
> fi
> done
>
To clean-up the directories, once done the broker starts without any issue.
Matthieu.
On Fri, May 17, 2024 at 2:45 PM Matthieu Patou <mpa...@rockset.com> wrote:
> Seems that actually running ./bin/kafka-reassign-partitions.sh to first
> generate and then execute to move the topic is working.
>
> I later realized that in a cluster where the metadata have been migrated
> to k-raft, the following command: ./bin/kafka-topics.sh --describe
> --bootstrap-server broker:9092
> Is showing you the topics and the topicID.
>
> I don't have more topics with topicID so I can't tell for sure if it will
> show if a topic doesn't have a topicID but it seems it should.
>
> Best.
>
> On Fri, May 17, 2024 at 10:33 AM Matthieu Patou <mpa...@rockset.com>
> wrote:
>
>> I'm trying to finish the migration of a dev cluster that used to use ZK.
>>
>> I finished the migration of Metadata from ZK to raft and it completed
>> fairly fast.
>> I did notice an error message:
>> * java.lang.RuntimeException: The log dir
>> Log(dir=/opt/kafka/data/topics/ingestperf.ingest-perf.ania001.3fd04d61e150165a-0,
>> topic=ingestperf.ingest-perf.ania001.3fd04d61e150165a, partition=0,
>> highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=190068)
>> does not have a topic ID, which is not allowed when running in KRaft mode.
>>
>> But it seemed that it was not causing the broker to crashloop and also it
>> seemed that the migration from ZK to Kafka worked.
>>
>> Now that I just restarted the broker after entering the migration of the
>> broker as described in
>> *"**Migrating brokers to KRaft" *in
>> https://kafka.apache.org/documentation/#kraft_zk_migration I'm seeing
>> this issue one more time but also the broker keeps on crashing.
>> I'm running Kafka 3.7.0 on the broker.
>> Looking at
>> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L1097
>> it
>> seems that there is something to take care of this situation but it seems
>> it's not working.
>>
>> I'm wondering how to get out of this situation.
>> Is there a way to assign a topic ID to topics that don't have one ?, is
>> there a way to list topics that don't have topic ID ?
>>
>> I'm thinking on this particular topic to use cruise control to migrate
>> the topic to another broker in the hope that it would force the creation of
>> a topic ID there (not sure tbh).
>>
>> Best.
>> Matthieu.
>>
>
