Robert V created KAFKA-7556:
-------------------------------
Summary: KafkaConsumer.beginningOffsets does not return actual
first offsets
Key: KAFKA-7556
URL: https://issues.apache.org/jira/browse/KAFKA-7556
Project: Kafka
Issue Type: Bug
Components: clients, consumer
Affects Versions: 1.0.0
Reporter: Robert V
h2. Description of the problem
The method `org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets`
claims in its Javadoc documentation that it would 'Get the first offset for the
given partitions.'.
I used it with a compacted topic, and it always returned offset 0 for all
partitions.
Not sure if using a compacted topic actually matters, but I'm enclosing this
information anyway.
Given a Kafka topic with retention set, and old log files being removed as a
result of that, the effective start offset of those partitions move further; it
simply will be greater than offset 0.
However, calling the `beginningOffsets` method always returns offset 0 as the
first offset.
In contrast, when the method
`org.apache.kafka.clients.consumer.KafkaConsumer.offsetsForTimes` is called
with a timestamp of 0L (UNIX epoch 1st Jan, 1970), it correctly returns the
effective start offsets for each partitions.
Output of using
`org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets`:
{code:java}
List((test.topic-87,Some(0)), (test.topic-54,Some(0)), (test.topic-21,Some(0)),
(test.topic-79,Some(0)), (test.topic-46,Some(0)), (test.topic-13,Some(0)),
(test.topic-70,Some(0)), (test.topic-37,Some(0)), (test.topic-12,Some(0)),
(test.topic-95,Some(0)), (test.topic-62,Some(0)), (test.topic-29,Some(0)),
(test.topic-4,Some(0)), (test.topic-88,Some(0)), (test.topic-55,Some(0)),
(test.topic-22,Some(0)), (test.topic-80,Some(0)), (test.topic-47,Some(0)),
(test.topic-14,Some(0)), (test.topic-71,Some(0)), (test.topic-38,Some(0)),
(test.topic-5,Some(0)), (test.topic-96,Some(0)), (test.topic-63,Some(0)),
(test.topic-30,Some(0)), (test.topic-56,Some(0)), (test.topic-23,Some(0)),
(test.topic-89,Some(0)), (test.topic-48,Some(0)), (test.topic-15,Some(0)),
(test.topic-81,Some(0)), (test.topic-72,Some(0)), (test.topic-39,Some(0)),
(test.topic-6,Some(0)), (test.topic-64,Some(0)), (test.topic-31,Some(0)),
(test.topic-97,Some(0)), (test.topic-24,Some(0)), (test.topic-90,Some(0)),
(test.topic-57,Some(0)), (test.topic-16,Some(0)), (test.topic-82,Some(0)),
(test.topic-49,Some(0)), (test.topic-40,Some(0)), (test.topic-7,Some(0)),
(test.topic-73,Some(0)), (test.topic-32,Some(0)), (test.topic-98,Some(0)),
(test.topic-65,Some(0)), (test.topic-91,Some(0)), (test.topic-58,Some(0)),
(test.topic-25,Some(0)), (test.topic-83,Some(0)), (test.topic-50,Some(0)),
(test.topic-17,Some(0)), (test.topic-8,Some(0)), (test.topic-74,Some(0)),
(test.topic-41,Some(0)), (test.topic-0,Some(0)), (test.topic-99,Some(0)),
(test.topic-66,Some(0)), (test.topic-33,Some(0)), (test.topic-92,Some(0)),
(test.topic-59,Some(0)), (test.topic-26,Some(0)), (test.topic-84,Some(0)),
(test.topic-51,Some(0)), (test.topic-18,Some(0)), (test.topic-75,Some(0)),
(test.topic-42,Some(0)), (test.topic-9,Some(0)), (test.topic-67,Some(0)),
(test.topic-34,Some(0)), (test.topic-1,Some(0)), (test.topic-85,Some(0)),
(test.topic-60,Some(0)), (test.topic-27,Some(0)), (test.topic-77,Some(0)),
(test.topic-52,Some(0)), (test.topic-19,Some(0)), (test.topic-76,Some(0)),
(test.topic-43,Some(0)), (test.topic-10,Some(0)), (test.topic-93,Some(0)),
(test.topic-68,Some(0)), (test.topic-35,Some(0)), (test.topic-2,Some(0)),
(test.topic-86,Some(0)), (test.topic-53,Some(0)), (test.topic-28,Some(0)),
(test.topic-78,Some(0)), (test.topic-45,Some(0)), (test.topic-20,Some(0)),
(test.topic-69,Some(0)), (test.topic-44,Some(0)), (test.topic-11,Some(0)),
(test.topic-94,Some(0)), (test.topic-61,Some(0)), (test.topic-36,Some(0)),
(test.topic-3,Some(0)))
{code}
Output of using
`org.apache.kafka.clients.consumer.KafkaConsumer.offsetsForTimes`:
{code:java}
List((test.topic-87,Some(289)), (test.topic-54,Some(45420)),
(test.topic-21,Some(63643)), (test.topic-79,Some(380750)),
(test.topic-46,Some(266379)), (test.topic-13,Some(98512)),
(test.topic-70,Some(118522)), (test.topic-37,Some(763)),
(test.topic-12,Some(61108)), (test.topic-95,Some(847647)),
(test.topic-62,Some(68299)), (test.topic-29,Some(548361)),
(test.topic-4,Some(144855)), (test.topic-88,Some(107831)),
(test.topic-55,Some(129241)), (test.topic-22,Some(563095)),
(test.topic-80,Some(534859)), (test.topic-47,Some(71608)),
(test.topic-14,Some(80962)), (test.topic-71,Some(83607)),
(test.topic-38,Some(166460)), (test.topic-5,Some(294307)),
(test.topic-96,Some(483237)), (test.topic-63,Some(441051)),
(test.topic-30,Some(151727)), (test.topic-56,Some(171045)),
(test.topic-23,Some(103736)), (test.topic-89,Some(276851)),
(test.topic-48,Some(696518)), (test.topic-15,Some(82049)),
(test.topic-81,Some(426329)), (test.topic-72,Some(76346)),
(test.topic-39,Some(39325)), (test.topic-6,Some(501911)),
(test.topic-64,Some(517845)), (test.topic-31,Some(68886)),
(test.topic-97,Some(378159)), (test.topic-24,Some(185533)),
(test.topic-90,Some(54403)), (test.topic-57,Some(154119)),
(test.topic-16,Some(91612)), (test.topic-82,Some(158657)),
(test.topic-49,Some(67356)), (test.topic-40,Some(133092)),
(test.topic-7,Some(178568)), (test.topic-73,Some(1897104)),
(test.topic-32,Some(291600)), (test.topic-98,Some(813386)),
(test.topic-65,Some(97431)), (test.topic-91,Some(302858)),
(test.topic-58,Some(416483)), (test.topic-25,Some(85250)),
(test.topic-83,Some(426855)), (test.topic-50,Some(151059)),
(test.topic-17,Some(71425)), (test.topic-8,Some(386542)),
(test.topic-74,Some(79210)), (test.topic-41,Some(84127)),
(test.topic-0,Some(56920)), (test.topic-99,Some(34036)),
(test.topic-66,Some(285773)), (test.topic-33,Some(627169)),
(test.topic-92,Some(67471)), (test.topic-59,Some(398275)),
(test.topic-26,Some(167519)), (test.topic-84,Some(1820740)),
(test.topic-51,Some(201574)), (test.topic-18,Some(447521)),
(test.topic-75,Some(101415)), (test.topic-42,Some(102338)),
(test.topic-9,Some(1012234)), (test.topic-67,Some(157516)),
(test.topic-34,Some(18917)), (test.topic-1,Some(114164)),
(test.topic-85,Some(59734)), (test.topic-60,Some(1395047)),
(test.topic-27,Some(567605)), (test.topic-77,Some(45907)),
(test.topic-52,Some(212559)), (test.topic-19,Some(98667)),
(test.topic-76,Some(304288)), (test.topic-43,Some(130635)),
(test.topic-10,Some(26730)), (test.topic-93,Some(545451)),
(test.topic-68,Some(160989)), (test.topic-35,Some(694191)),
(test.topic-2,Some(79696)), (test.topic-86,Some(142042)),
(test.topic-53,Some(44083)), (test.topic-28,Some(97500)),
(test.topic-78,Some(95872)), (test.topic-45,Some(346044)),
(test.topic-20,Some(166317)), (test.topic-69,Some(143188)),
(test.topic-44,Some(870214)), (test.topic-11,Some(182125)),
(test.topic-94,Some(1022343)), (test.topic-61,Some(1204397)),
(test.topic-36,Some(172640)), (test.topic-3,Some(98632)))
{code}
h2. Proposed fix
* If the intention is to return what the beginning offset ever was, the
Javadoc documentation should be explicit about it and another method should be
added to query the effective start offsets.
* Alternatively, the method should return the actual start offset of the
partitions at the time of calling this method. Which, for topics with retention
policy set, is not equal to what the start offsets were before removing log
files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
