[ https://issues.apache.org/jira/browse/KAFKA-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665428#comment-16665428 ]
Mayuresh Gharat commented on KAFKA-7556: ---------------------------------------- Hi [~rob_v], This seems like a bug to me. Let me take a look, now. Assigning this ticket to myself for now. Will change it to unassign (so others can pick this up), if I cannot get to this done in timely manner. Thanks, Mayuresh > KafkaConsumer.beginningOffsets does not return actual first offsets > ------------------------------------------------------------------- > > Key: KAFKA-7556 > URL: https://issues.apache.org/jira/browse/KAFKA-7556 > Project: Kafka > Issue Type: Bug > Components: clients, consumer > Affects Versions: 1.0.0 > Reporter: Robert V > Priority: Critical > Labels: documentation, usability > Fix For: 2.2.0 > > > h2. Description of the problem > The method `org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets` > claims in its Javadoc documentation that it would 'Get the first offset for > the given partitions.'. > I used it with a compacted topic, and it always returned offset 0 for all > partitions. > Not sure if using a compacted topic actually matters, but I'm enclosing this > information anyway. > Given a Kafka topic with retention set, and old log files being removed as a > result of that, the effective start offset of those partitions move further; > it simply will be greater than offset 0. > However, calling the `beginningOffsets` method always returns offset 0 as the > first offset. > In contrast, when the method > `org.apache.kafka.clients.consumer.KafkaConsumer.offsetsForTimes` is called > with a timestamp of 0L (UNIX epoch 1st Jan, 1970), it correctly returns the > effective start offsets for each partitions. > Output of using > `org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets`: > {code:java} > {test.topic-87=0, test.topic-54=0, test.topic-21=0, test.topic-79=0, > test.topic-46=0, test.topic-13=0, test.topic-70=0, test.topic-37=0, > test.topic-12=0, test.topic-95=0, test.topic-62=0, test.topic-29=0, > test.topic-4=0, test.topic-88=0, test.topic-55=0, test.topic-22=0, > test.topic-80=0, test.topic-47=0, test.topic-14=0, test.topic-71=0, > test.topic-38=0, test.topic-5=0, test.topic-96=0, test.topic-63=0, > test.topic-30=0, test.topic-56=0, test.topic-23=0, test.topic-89=0, > test.topic-48=0, test.topic-15=0, test.topic-81=0, test.topic-72=0, > test.topic-39=0, test.topic-6=0, test.topic-64=0, test.topic-31=0, > test.topic-97=0, test.topic-24=0, test.topic-90=0, test.topic-57=0, > test.topic-16=0, test.topic-82=0, test.topic-49=0, test.topic-40=0, > test.topic-7=0, test.topic-73=0, test.topic-32=0, test.topic-98=0, > test.topic-65=0, test.topic-91=0, test.topic-58=0, test.topic-25=0, > test.topic-83=0, test.topic-50=0, test.topic-17=0, test.topic-8=0, > test.topic-74=0, test.topic-41=0, test.topic-0=0, test.topic-99=0, > test.topic-66=0, test.topic-33=0, test.topic-92=0, test.topic-59=0, > test.topic-26=0, test.topic-84=0, test.topic-51=0, test.topic-18=0, > test.topic-75=0, test.topic-42=0, test.topic-9=0, test.topic-67=0, > test.topic-34=0, test.topic-1=0, test.topic-85=0, test.topic-60=0, > test.topic-27=0, test.topic-77=0, test.topic-52=0, test.topic-19=0, > test.topic-76=0, test.topic-43=0, test.topic-10=0, test.topic-93=0, > test.topic-68=0, test.topic-35=0, test.topic-2=0, test.topic-86=0, > test.topic-53=0, test.topic-28=0, test.topic-78=0, test.topic-45=0, > test.topic-20=0, test.topic-69=0, test.topic-44=0, test.topic-11=0, > test.topic-94=0, test.topic-61=0, test.topic-36=0, test.topic-3=0} > {code} > Output of using > `org.apache.kafka.clients.consumer.KafkaConsumer.offsetsForTimes`: > {code:java} > {test.topic-87=(timestamp=1511264434285, offset=289), > test.topic-54=(timestamp=1511265134993, offset=45420), > test.topic-21=(timestamp=1511265534207, offset=63643), > test.topic-79=(timestamp=1511270338275, offset=380750), > test.topic-46=(timestamp=1511266883588, offset=266379), > test.topic-13=(timestamp=1511265900538, offset=98512), > test.topic-70=(timestamp=1511266972452, offset=118522), > test.topic-37=(timestamp=1511264396370, offset=763), > test.topic-12=(timestamp=1511265504886, offset=61108), > test.topic-95=(timestamp=1511289492800, offset=847647), > test.topic-62=(timestamp=1511265831298, offset=68299), > test.topic-29=(timestamp=1511278767417, offset=548361), > test.topic-4=(timestamp=1511269316679, offset=144855), > test.topic-88=(timestamp=1511265608468, offset=107831), > test.topic-55=(timestamp=1511267449288, offset=129241), > test.topic-22=(timestamp=1511283134114, offset=563095), > test.topic-80=(timestamp=1511277334877, offset=534859), > test.topic-47=(timestamp=1511265530689, offset=71608), > test.topic-14=(timestamp=1511266308829, offset=80962), > test.topic-71=(timestamp=1511265474740, offset=83607), > test.topic-38=(timestamp=1511268268259, offset=166460), > test.topic-5=(timestamp=1511276243850, offset=294307), > test.topic-96=(timestamp=1511276749138, offset=483237), > test.topic-63=(timestamp=1511276798188, offset=441051), > test.topic-30=(timestamp=1511269265206, offset=151727), > test.topic-56=(timestamp=1511267813861, offset=171045), > test.topic-23=(timestamp=1511268644790, offset=103736), > test.topic-89=(timestamp=1511270771269, offset=276851), > test.topic-48=(timestamp=1511279335687, offset=696518), > test.topic-15=(timestamp=1511265886647, offset=82049), > test.topic-81=(timestamp=1511272770367, offset=426329), > test.topic-72=(timestamp=1511265783620, offset=76346), > test.topic-39=(timestamp=1511265444819, offset=39325), > test.topic-6=(timestamp=1511343902830, offset=501911), > test.topic-64=(timestamp=1511276754029, offset=517845), > test.topic-31=(timestamp=1511265905957, offset=68886), > test.topic-97=(timestamp=1511271052868, offset=378159), > test.topic-24=(timestamp=1511268471118, offset=185533), > test.topic-90=(timestamp=1511265280669, offset=54403), > test.topic-57=(timestamp=1511266299095, offset=154119), > test.topic-16=(timestamp=1511266140680, offset=91612), > test.topic-82=(timestamp=1511266586862, offset=158657), > test.topic-49=(timestamp=1511265550828, offset=67356), > test.topic-40=(timestamp=1511266374225, offset=133092), > test.topic-7=(timestamp=1511270077779, offset=178568), > test.topic-73=(timestamp=1511324184866, offset=1897104), > test.topic-32=(timestamp=1511269989097, offset=291600), > test.topic-98=(timestamp=1511282897007, offset=813386), > test.topic-65=(timestamp=1511265848788, offset=97431), > test.topic-91=(timestamp=1511270007091, offset=302858), > test.topic-58=(timestamp=1511273273743, offset=416483), > test.topic-25=(timestamp=1511266304470, offset=85250), > test.topic-83=(timestamp=1511272177062, offset=426855), > test.topic-50=(timestamp=1511266726637, offset=151059), > test.topic-17=(timestamp=1511265892637, offset=71425), > test.topic-8=(timestamp=1511276315670, offset=386542), > test.topic-74=(timestamp=1511265671808, offset=79210), > test.topic-41=(timestamp=1511265957561, offset=84127), > test.topic-0=(timestamp=1511265529344, offset=56920), > test.topic-99=(timestamp=1511265343408, offset=34036), > test.topic-66=(timestamp=1511268683052, offset=285773), > test.topic-33=(timestamp=1511292268724, offset=627169), > test.topic-92=(timestamp=1511265745483, offset=67471), > test.topic-59=(timestamp=1511275909747, offset=398275), > test.topic-26=(timestamp=1511268821082, offset=167519), > test.topic-84=(timestamp=1511320800595, offset=1820740), > test.topic-51=(timestamp=1511267699931, offset=201574), > test.topic-18=(timestamp=1511277137591, offset=447521), > test.topic-75=(timestamp=1511266718159, offset=101415), > test.topic-42=(timestamp=1511268280357, offset=102338), > test.topic-9=(timestamp=1511321331900, offset=1012234), > test.topic-67=(timestamp=1511266755881, offset=157516), > test.topic-34=(timestamp=1511265028871, offset=18917), > test.topic-1=(timestamp=1511269548372, offset=114164), > test.topic-85=(timestamp=1511265426041, offset=59734), > test.topic-60=(timestamp=1511299615444, offset=1395047), > test.topic-27=(timestamp=1511284039796, offset=567605), > test.topic-77=(timestamp=1511265412189, offset=45907), > test.topic-52=(timestamp=1511268603935, offset=212559), > test.topic-19=(timestamp=1511266992878, offset=98667), > test.topic-76=(timestamp=1511269937877, offset=304288), > test.topic-43=(timestamp=1511267795292, offset=130635), > test.topic-10=(timestamp=1511265000373, offset=26730), > test.topic-93=(timestamp=1511275532751, offset=545451), > test.topic-68=(timestamp=1511266797346, offset=160989), > test.topic-35=(timestamp=1511279364499, offset=694191), > test.topic-2=(timestamp=1511270517080, offset=79696), > test.topic-86=(timestamp=1511266816638, offset=142042), > test.topic-53=(timestamp=1511265524531, offset=44083), > test.topic-28=(timestamp=1511266431654, offset=97500), > test.topic-78=(timestamp=1511266431145, offset=95872), > test.topic-45=(timestamp=1511273274002, offset=346044), > test.topic-20=(timestamp=1511268405607, offset=166317), > test.topic-69=(timestamp=1511266953570, offset=143188), > test.topic-44=(timestamp=1511289270173, offset=870214), > test.topic-11=(timestamp=1511271446913, offset=182125), > test.topic-94=(timestamp=1511290978907, offset=1022343), > test.topic-61=(timestamp=1511303529222, offset=1204397), > test.topic-36=(timestamp=1511266872111, offset=172640), > test.topic-3=(timestamp=1511266312670, offset=98632)} > {code} > h2. Proposed fix > * If the intention is to return what the beginning offset ever was, the > Javadoc documentation should be explicit about it and another method should > be added to query the effective start offsets. > * Alternatively, the method should return the actual start offset of the > partitions at the time of calling this method. Which, for topics with > retention policy set, is not equal to what the start offsets were before > removing log files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)