Christopher Auston created KAFKA-15006: ------------------------------------------
Summary: kafka-get-offsets.sh returns 0 for earliest offset on a compacted topic Key: KAFKA-15006 URL: https://issues.apache.org/jira/browse/KAFKA-15006 Project: Kafka Issue Type: Bug Affects Versions: 3.4.0 Reporter: Christopher Auston After some compactions and when the 00000000000000000000.log is empty, it seems like kafka-get-offsets.sh (Admin client?) should return a non-zero offset for the earliest offset? A simple, compacted topic: {code:java} kraft-combined-logs % kafka-topics.sh --bootstrap-server :9092 -topic offtest -describeTopic: offtest TopicId: pwmz5qPGQsyo3006M_VBqw PartitionCount: 1 ReplicationFactor: 1 Configs: cleanup.policy=compact,segment.bytes=16384 Topic: offtest Partition: 0 Leader: 1 Replicas: 1 Isr: 1{code} Trying to get the earliest offset after some compactions returns 0 which is not where a consumer starts with earliest (also the 000 log segment is empty): {code:java} kraft-combined-logs % kafka-get-offsets.sh --bootstrap-server :9092 -topic offtest -time -2 offtest:0:0 kraft-combined-logs % kcat -b :9092 -C -t offtest -c1 -J {"topic":"offtest","partition":0,"offset":161,"tstype":"create","ts":1684347565956,"broker":1,"key":"key1","payload":" value 34"} kraft-combined-logs % ls -l offtest-0 total 41024 -rw-r--r-- 1 ca9898 staff 0 May 17 14:16 00000000000000000000.index -rw-r--r-- 1 ca9898 staff 0 May 17 14:16 00000000000000000000.log -rw-r--r-- 1 ca9898 staff 0 May 17 14:16 00000000000000000000.timeindex -rw-r--r-- 1 ca9898 staff 0 May 17 14:19 00000000000000000081.index -rw-r--r-- 1 ca9898 staff 202 May 17 14:19 00000000000000000081.log -rw-r--r-- 1 ca9898 staff 10 May 17 14:16 00000000000000000081.snapshot -rw-r--r-- 1 ca9898 staff 12 May 17 14:19 00000000000000000081.timeindex -rw-r--r-- 1 ca9898 staff 10485760 May 17 14:19 00000000000000000162.index -rw-r--r-- 1 ca9898 staff 6060 May 17 14:19 00000000000000000162.log -rw-r--r-- 1 ca9898 staff 10 May 17 14:19 00000000000000000162.snapshot -rw-r--r-- 1 ca9898 staff 10485756 May 17 14:19 00000000000000000162.timeindex -rw-r--r-- 1 ca9898 staff 8 May 17 13:56 leader-epoch-checkpoint -rw-r--r-- 1 ca9898 staff 43 May 17 13:56 partition.metadata {code} Server console log: {code:java} [2023-05-17 13:50:52,461] INFO [SocketServer listenerType=BROKER, nodeId=1] Enabling request processing. (kafka.network.SocketServer) [2023-05-17 13:50:52,496] INFO [BrokerLifecycleManager id=1] The broker has been unfenced. Transitioning from RECOVERY to RUNNING. (kafka.server.BrokerLifecycleManager) [2023-05-17 13:50:52,496] INFO [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer) [2023-05-17 13:50:52,496] INFO Kafka version: 3.4.0 (org.apache.kafka.common.utils.AppInfoParser) [2023-05-17 13:50:52,500] INFO Kafka commitId: 2e1947d240607d53 (org.apache.kafka.common.utils.AppInfoParser) [2023-05-17 13:50:52,500] INFO Kafka startTimeMs: 1684345852496 (org.apache.kafka.common.utils.AppInfoParser) [2023-05-17 13:50:52,501] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer) [2023-05-17 13:56:18,521] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions Set(offtest-0) (kafka.server.ReplicaFetcherManager) [2023-05-17 13:56:18,530] INFO [LogLoader partition=offtest-0, dir=/Users/ca9898/kraft-combined-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.UnifiedLog$) [2023-05-17 13:56:18,531] INFO Created log for partition offtest-0 in /Users/ca9898/kraft-combined-logs/offtest-0 with properties {} (kafka.log.LogManager) [2023-05-17 13:56:18,536] INFO [Partition offtest-0 broker=1] No checkpointed highwatermark is found for partition offtest-0 (kafka.cluster.Partition) [2023-05-17 13:56:18,536] INFO [Partition offtest-0 broker=1] Log loaded for partition offtest-0 with initial high watermark 0 (kafka.cluster.Partition) [2023-05-17 14:06:25,254] INFO [DynamicConfigPublisher nodeType=broker id=1] Updating topic offtest with new configuration : cleanup.policy -> compact,segment.bytes -> 16384 (kafka.server.metadata.DynamicConfigPublisher) [2023-05-17 14:16:30,990] INFO [BrokerToControllerChannelManager broker=1 name=forwarding] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient) [2023-05-17 14:16:53,142] INFO [LocalLog partition=offtest-0, dir=/Users/ca9898/kraft-combined-logs] Rolled new log segment at offset 81 in 2 ms. (kafka.log.LocalLog) [2023-05-17 14:16:53,147] INFO [ProducerStateManager partition=offtest-0] Wrote producer snapshot at offset 81 with 0 producer ids in 5 ms. (kafka.log.ProducerStateManager) [2023-05-17 14:18:08,020] INFO [UnifiedLog partition=offtest-0, dir=/Users/ca9898/kraft-combined-logs] Deleting segment files LogSegment(baseOffset=0, size=16362, lastModifiedTime=1684347413121, largestRecordTimestamp=Some(1684347413113)) (kafka.log.LocalLog$) [2023-05-17 14:18:08,028] INFO Deleted log /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.log.deleted. (kafka.log.LogSegment) [2023-05-17 14:18:08,033] INFO Deleted offset index /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.index.deleted. (kafka.log.LogSegment) [2023-05-17 14:18:08,033] INFO Deleted time index /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment) [2023-05-17 14:19:25,983] INFO [LocalLog partition=offtest-0, dir=/Users/ca9898/kraft-combined-logs] Rolled new log segment at offset 162 in 1 ms. (kafka.log.LocalLog) [2023-05-17 14:19:25,987] INFO [ProducerStateManager partition=offtest-0] Wrote producer snapshot at offset 162 with 0 producer ids in 5 ms. (kafka.log.ProducerStateManager) [2023-05-17 14:20:38,102] INFO [UnifiedLog partition=offtest-0, dir=/Users/ca9898/kraft-combined-logs] Deleting segment files LogSegment(baseOffset=0, size=202, lastModifiedTime=1684347413121, largestRecordTimestamp=Some(1684347413113)) (kafka.log.LocalLog$) [2023-05-17 14:20:38,104] INFO Deleted log /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.log.deleted. (kafka.log.LogSegment) [2023-05-17 14:20:38,104] INFO Deleted offset index /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.index.deleted. (kafka.log.LogSegment) [2023-05-17 14:20:38,105] INFO Deleted time index /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment) [2023-05-17 14:20:38,116] INFO [UnifiedLog partition=offtest-0, dir=/Users/ca9898/kraft-combined-logs] Deleting segment files LogSegment(baseOffset=81, size=16362, lastModifiedTime=1684347565963, largestRecordTimestamp=Some(1684347565956)) (kafka.log.LocalLog$) [2023-05-17 14:20:38,117] INFO Deleted log /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000081.log.deleted. (kafka.log.LogSegment) [2023-05-17 14:20:38,117] INFO Deleted offset index /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000081.index.deleted. (kafka.log.LogSegment) [2023-05-17 14:20:38,117] INFO Deleted time index /Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000081.timeindex.deleted. (kafka.log.LogSegment) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)