Greetings, Kafka specialists
A strange issue in my Kafka instance that blocked my for a couple of days.
1. Cannot produce event to [topic-test] due to a KAFKA_STORAGE_ERROR
2. It seems to me the log segment file is not having any problem.
I can dump log, index, timeindex with kafka-dump-log without seeing any error.
3. No error when I tried to produce events to other topics. Kafka log
files are in a same disk partition.
4. Restart Kafka instance and zookeeper instance did not help.
5. I cannot find useful information about the error from server.log with
TRACE level.
Could you please let me know whether similar issue ever happened before?
Where shall I go to dig further and continue my investigation?
Thanks a lot!
Peter
The kafka client log
kafka-console-producer --bootstrap-server 135.251.236.162:9092 --topic
topic-test
>[2022-04-28 11:12:10,925] WARN [Producer clientId=console-producer] Got error
>produce response with correlation id 5 on topic-partition topic-test-0,
>retrying (2 attempts left). Error: KAFKA_STORAGE_ERROR
>(org.apache.kafka.clients.producer.internals.Sender)
[2022-04-28 11:12:10,925] WARN [Producer clientId=console-producer] Received
invalid metadata error in produce request on partition topic-test-0 due to
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to
access log file on the disk.. Going to request metadata update now
(org.apache.kafka.clients.producer.internals.Sender)
[2022-04-28 11:12:11,024] WARN [Producer clientId=console-producer] Got error
produce response with correlation id 7 on topic-partition topic-test-0,
retrying (1 attempts left). Error: KAFKA_STORAGE_ERROR
(org.apache.kafka.clients.producer.internals.Sender)
[2022-04-28 11:12:11,024] WARN [Producer clientId=console-producer] Received
invalid metadata error in produce request on partition topic-test-0 due to
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to
access log file on the disk.. Going to request metadata update now
(org.apache.kafka.clients.producer.internals.Sender)
[2022-04-28 11:12:11,127] WARN [Producer clientId=console-producer] Got error
produce response with correlation id 9 on topic-partition topic-test-0,
retrying (0 attempts left). Error: KAFKA_STORAGE_ERROR
(org.apache.kafka.clients.producer.internals.Sender)
[2022-04-28 11:12:11,127] WARN [Producer clientId=console-producer] Received
invalid metadata error in produce request on partition topic-test-0 due to
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to
access log file on the disk.. Going to request metadata update now
(org.apache.kafka.clients.producer.internals.Sender)
[2022-04-28 11:12:11,231] ERROR Error when sending message to topic topic-test
with key: null, value: 0 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to
access log file on the disk.
[2022-04-28 11:12:11,233] WARN [Producer clientId=console-producer] Received
invalid metadata error in produce request on partition topic-test-0 due to
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to
access log file on the disk.. Going to request metadata update now
(org.apache.kafka.clients.producer.internals.Sender)
The server.log
[2022-04-27 07:41:51,203] TRACE [KafkaApi-0] Handling
request:RequestHeader(apiKey=METADATA, apiVersion=11,
clientId=console-producer, correlationId=9) --
MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='topic-test')], allowAutoTopicCreation=true,
includeClusterAuthorizedOperations=false,
includeTopicAuthorizedOperations=false) from connection
135.251.236.162:9092-135.251.236.162:44194-2;securityProtocol:PLAINTEXT,principal:User:ANONYMOUS
(kafka.server.KafkaApis)
[2022-04-27 07:41:51,203] TRACE [KafkaApi-0] Sending topic metadata
MetadataResponseTopic(errorCode=0, name='topic-tst',
topicId=mgS7D7-9RZSgeEUJ3XXErw, isInternal=false,
partitions=[MetadataResponsePartition(errorCode=0, partitionIndex=0,
leaderId=0, leaderEpoch=0, replicaNodes=[0], isrNodes=[0],
offlineReplicas=[])], topicAuthorizedOperations=-2147483648) and brokers
baijin162-vnfprov:9092 (id: 0 rack: null) for correlation id 9 to client
console-producer (kafka.server.KafkaApis)
[2022-04-27 07:41:51,297] TRACE [KafkaApi-0] Handling
request:RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=console-producer,
correlationId=10) -- {acks=1,timeout=1500,partitionSizes=[topic-test-0=81]}
from connection
135.251.236.162:9092-135.251.236.162:44194-2;securityProtocol:PLAINTEXT,principal:User:ANONYMOUS
(kafka.server.KafkaApis)
[2022-04-27 07:41:51,297] TRACE [ReplicaManager broker=0] Append
[HashMap(topic-test-0 -> MemoryRecords(size=81,
buffer=java.nio.HeapByteBuffer[pos=0 lim=81 cap=84]))] to local log
(kafka.server.ReplicaManager)
[2022-04-27 07:41:51,297] DEBUG [KafkaApi-0] Produce request with correlation
id 10 from client console-producer on partition topic-test-0 failed due to
org.apache.kafka.common.errors.KafkaStorageException (kafka.server.KafkaApis)
