I have found out what was the issue -- it was bug in the library. Thanks everyone for help.
On Tue, Apr 21, 2015 at 9:02 PM, Andrei <[email protected]> wrote: > @Piotr I can confirm that in native (Java/Scala) library returned message > offsets start with the one specified in request, not the next one. Note > sure about underlying protocol, though. > > As long as you also know Python, I would suggest you looking at > kafka-python [1] library that is pretty readable and has already solved > most issues you will encounter. > > [1]: https://github.com/mumrah/kafka-python > > > > On Tue, Apr 21, 2015 at 6:46 PM, Piotr Husiatyński <[email protected]> wrote: > >> I'm using Go with kafka library** and sending fetch request as >> described in wire protocol documentation returns messages off by one. >> Are you sure that code that you wrote is not sending fetch request >> with offset being one less than you pass? >> >> ** https://github.com/optiopay/kafka >> >> On Tue, Apr 21, 2015 at 4:54 PM, Manoj Khangaonkar >> <[email protected]> wrote: >> > Hi, >> > >> > I have used code like >> > >> > FetchRequest req = new FetchRequestBuilder() >> > .clientId(clientname) >> > .addFetch(topic, partition, offset_in_partition, >> 100000).build(); >> > FetchResponse fetchResponse = consumer.fetch(req); >> > >> > And it returns the message that I was looking for. >> > >> > The offset_in_partion is a value I get by calling >> > >> > offset_in_partition = messageAndOffset.nextOffset(); // call this on a >> > previous message >> > >> > and saving it somewhere for future use. >> > >> > The only caveat is that fetch returns a block of messages which means I >> > might >> > get messages with lower offset that I just need to ignore. >> > >> > regards >> > >> > >> > On Tue, Apr 21, 2015 at 6:47 AM, Piotr Husiatyński <[email protected]> >> wrote: >> > >> >> What do you mean that message has expired? >> >> >> >> If I will do fetch request with offset 4, I will receive first message >> >> with offset 5. I think fetch is always excluding requested offset, >> >> while documentation is saying it will be included in response. >> >> >> >> On Tue, Apr 21, 2015 at 3:35 PM, Manoj Khangaonkar >> >> <[email protected]> wrote: >> >> > Hi, >> >> > >> >> > I suspect If some message from the given offset have expired, then >> they >> >> > will not be returned. >> >> > >> >> > regards >> >> > >> >> > On Tue, Apr 21, 2015 at 5:14 AM, Piotr Husiatyński <[email protected]> >> >> wrote: >> >> > >> >> >> According to documentation, sending fetch request with offset value >> >> >> result in messages starting with given offset (including) and >> greater. >> >> >> >> >> >> >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI >> >> >> >> >> >> > Logically one specifies the topics, partitions, and starting >> offset at >> >> >> which to begin the fetch and gets back a chunk of messages. In >> general, >> >> the >> >> >> return messages will have offsets larger than or equal to the >> starting >> >> >> offset >> >> >> >> >> >> I'm sending fetch request with offset 5 and first message I'm getting >> >> >> is offset 6. Am I doing something wrong or documentation has to be >> >> >> fixed? >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > http://khangaonkar.blogspot.com/ >> >> >> > >> > >> > >> > -- >> > http://khangaonkar.blogspot.com/ >>
