Christopher Auston created KAFKA-15006:
------------------------------------------
Summary: kafka-get-offsets.sh returns 0 for earliest offset on a
compacted topic
Key: KAFKA-15006
URL: https://issues.apache.org/jira/browse/KAFKA-15006
Project: Kafka
Issue Type: Bug
Affects Versions: 3.4.0
Reporter: Christopher Auston
After some compactions and when the 00000000000000000000.log is empty, it seems
like kafka-get-offsets.sh (Admin client?) should return a non-zero offset for
the earliest offset?
A simple, compacted topic:
{code:java}
kraft-combined-logs % kafka-topics.sh --bootstrap-server :9092 -topic offtest
-describeTopic: offtest TopicId: pwmz5qPGQsyo3006M_VBqw PartitionCount: 1
ReplicationFactor: 1 Configs: cleanup.policy=compact,segment.bytes=16384
Topic: offtest Partition: 0 Leader: 1 Replicas: 1 Isr: 1{code}
Trying to get the earliest offset after some compactions returns 0 which is not
where a consumer starts with earliest (also the 000 log segment is empty):
{code:java}
kraft-combined-logs % kafka-get-offsets.sh --bootstrap-server :9092 -topic
offtest -time -2
offtest:0:0
kraft-combined-logs % kcat -b :9092 -C -t offtest -c1 -J
{"topic":"offtest","partition":0,"offset":161,"tstype":"create","ts":1684347565956,"broker":1,"key":"key1","payload":"
value 34"}
kraft-combined-logs % ls -l offtest-0
total 41024
-rw-r--r-- 1 ca9898 staff 0 May 17 14:16 00000000000000000000.index
-rw-r--r-- 1 ca9898 staff 0 May 17 14:16 00000000000000000000.log
-rw-r--r-- 1 ca9898 staff 0 May 17 14:16
00000000000000000000.timeindex
-rw-r--r-- 1 ca9898 staff 0 May 17 14:19 00000000000000000081.index
-rw-r--r-- 1 ca9898 staff 202 May 17 14:19 00000000000000000081.log
-rw-r--r-- 1 ca9898 staff 10 May 17 14:16 00000000000000000081.snapshot
-rw-r--r-- 1 ca9898 staff 12 May 17 14:19
00000000000000000081.timeindex
-rw-r--r-- 1 ca9898 staff 10485760 May 17 14:19 00000000000000000162.index
-rw-r--r-- 1 ca9898 staff 6060 May 17 14:19 00000000000000000162.log
-rw-r--r-- 1 ca9898 staff 10 May 17 14:19 00000000000000000162.snapshot
-rw-r--r-- 1 ca9898 staff 10485756 May 17 14:19
00000000000000000162.timeindex
-rw-r--r-- 1 ca9898 staff 8 May 17 13:56 leader-epoch-checkpoint
-rw-r--r-- 1 ca9898 staff 43 May 17 13:56 partition.metadata
{code}
Server console log:
{code:java}
[2023-05-17 13:50:52,461] INFO [SocketServer listenerType=BROKER, nodeId=1]
Enabling request processing. (kafka.network.SocketServer)
[2023-05-17 13:50:52,496] INFO [BrokerLifecycleManager id=1] The broker has
been unfenced. Transitioning from RECOVERY to RUNNING.
(kafka.server.BrokerLifecycleManager)
[2023-05-17 13:50:52,496] INFO [BrokerServer id=1] Transition from STARTING to
STARTED (kafka.server.BrokerServer)
[2023-05-17 13:50:52,496] INFO Kafka version: 3.4.0
(org.apache.kafka.common.utils.AppInfoParser)
[2023-05-17 13:50:52,500] INFO Kafka commitId: 2e1947d240607d53
(org.apache.kafka.common.utils.AppInfoParser)
[2023-05-17 13:50:52,500] INFO Kafka startTimeMs: 1684345852496
(org.apache.kafka.common.utils.AppInfoParser)
[2023-05-17 13:50:52,501] INFO [KafkaRaftServer nodeId=1] Kafka Server started
(kafka.server.KafkaRaftServer)
[2023-05-17 13:56:18,521] INFO [ReplicaFetcherManager on broker 1] Removed
fetcher for partitions Set(offtest-0) (kafka.server.ReplicaFetcherManager)
[2023-05-17 13:56:18,530] INFO [LogLoader partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Loading producer state till offset 0
with message format version 2 (kafka.log.UnifiedLog$)
[2023-05-17 13:56:18,531] INFO Created log for partition offtest-0 in
/Users/ca9898/kraft-combined-logs/offtest-0 with properties {}
(kafka.log.LogManager)
[2023-05-17 13:56:18,536] INFO [Partition offtest-0 broker=1] No checkpointed
highwatermark is found for partition offtest-0 (kafka.cluster.Partition)
[2023-05-17 13:56:18,536] INFO [Partition offtest-0 broker=1] Log loaded for
partition offtest-0 with initial high watermark 0 (kafka.cluster.Partition)
[2023-05-17 14:06:25,254] INFO [DynamicConfigPublisher nodeType=broker id=1]
Updating topic offtest with new configuration : cleanup.policy ->
compact,segment.bytes -> 16384 (kafka.server.metadata.DynamicConfigPublisher)
[2023-05-17 14:16:30,990] INFO [BrokerToControllerChannelManager broker=1
name=forwarding] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-05-17 14:16:53,142] INFO [LocalLog partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Rolled new log segment at offset 81 in 2
ms. (kafka.log.LocalLog)
[2023-05-17 14:16:53,147] INFO [ProducerStateManager partition=offtest-0] Wrote
producer snapshot at offset 81 with 0 producer ids in 5 ms.
(kafka.log.ProducerStateManager)
[2023-05-17 14:18:08,020] INFO [UnifiedLog partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Deleting segment files
LogSegment(baseOffset=0, size=16362, lastModifiedTime=1684347413121,
largestRecordTimestamp=Some(1684347413113)) (kafka.log.LocalLog$)
[2023-05-17 14:18:08,028] INFO Deleted log
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.log.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:18:08,033] INFO Deleted offset index
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.index.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:18:08,033] INFO Deleted time index
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.timeindex.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:19:25,983] INFO [LocalLog partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Rolled new log segment at offset 162 in
1 ms. (kafka.log.LocalLog)
[2023-05-17 14:19:25,987] INFO [ProducerStateManager partition=offtest-0] Wrote
producer snapshot at offset 162 with 0 producer ids in 5 ms.
(kafka.log.ProducerStateManager)
[2023-05-17 14:20:38,102] INFO [UnifiedLog partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Deleting segment files
LogSegment(baseOffset=0, size=202, lastModifiedTime=1684347413121,
largestRecordTimestamp=Some(1684347413113)) (kafka.log.LocalLog$)
[2023-05-17 14:20:38,104] INFO Deleted log
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.log.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:20:38,104] INFO Deleted offset index
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.index.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:20:38,105] INFO Deleted time index
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000000.timeindex.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:20:38,116] INFO [UnifiedLog partition=offtest-0,
dir=/Users/ca9898/kraft-combined-logs] Deleting segment files
LogSegment(baseOffset=81, size=16362, lastModifiedTime=1684347565963,
largestRecordTimestamp=Some(1684347565956)) (kafka.log.LocalLog$)
[2023-05-17 14:20:38,117] INFO Deleted log
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000081.log.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:20:38,117] INFO Deleted offset index
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000081.index.deleted.
(kafka.log.LogSegment)
[2023-05-17 14:20:38,117] INFO Deleted time index
/Users/ca9898/kraft-combined-logs/offtest-0/00000000000000000081.timeindex.deleted.
(kafka.log.LogSegment) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
