@Piotr I can confirm that in native (Java/Scala) library returned message offsets start with the one specified in request, not the next one. Note sure about underlying protocol, though.
As long as you also know Python, I would suggest you looking at kafka-python [1] library that is pretty readable and has already solved most issues you will encounter. [1]: https://github.com/mumrah/kafka-python On Tue, Apr 21, 2015 at 6:46 PM, Piotr Husiatyński <p...@optiopay.com> wrote: > I'm using Go with kafka library** and sending fetch request as > described in wire protocol documentation returns messages off by one. > Are you sure that code that you wrote is not sending fetch request > with offset being one less than you pass? > > ** https://github.com/optiopay/kafka > > On Tue, Apr 21, 2015 at 4:54 PM, Manoj Khangaonkar > <khangaon...@gmail.com> wrote: > > Hi, > > > > I have used code like > > > > FetchRequest req = new FetchRequestBuilder() > > .clientId(clientname) > > .addFetch(topic, partition, offset_in_partition, > 100000).build(); > > FetchResponse fetchResponse = consumer.fetch(req); > > > > And it returns the message that I was looking for. > > > > The offset_in_partion is a value I get by calling > > > > offset_in_partition = messageAndOffset.nextOffset(); // call this on a > > previous message > > > > and saving it somewhere for future use. > > > > The only caveat is that fetch returns a block of messages which means I > > might > > get messages with lower offset that I just need to ignore. > > > > regards > > > > > > On Tue, Apr 21, 2015 at 6:47 AM, Piotr Husiatyński <p...@optiopay.com> > wrote: > > > >> What do you mean that message has expired? > >> > >> If I will do fetch request with offset 4, I will receive first message > >> with offset 5. I think fetch is always excluding requested offset, > >> while documentation is saying it will be included in response. > >> > >> On Tue, Apr 21, 2015 at 3:35 PM, Manoj Khangaonkar > >> <khangaon...@gmail.com> wrote: > >> > Hi, > >> > > >> > I suspect If some message from the given offset have expired, then > they > >> > will not be returned. > >> > > >> > regards > >> > > >> > On Tue, Apr 21, 2015 at 5:14 AM, Piotr Husiatyński <p...@optiopay.com> > >> wrote: > >> > > >> >> According to documentation, sending fetch request with offset value > >> >> result in messages starting with given offset (including) and > greater. > >> >> > >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI > >> >> > >> >> > Logically one specifies the topics, partitions, and starting > offset at > >> >> which to begin the fetch and gets back a chunk of messages. In > general, > >> the > >> >> return messages will have offsets larger than or equal to the > starting > >> >> offset > >> >> > >> >> I'm sending fetch request with offset 5 and first message I'm getting > >> >> is offset 6. Am I doing something wrong or documentation has to be > >> >> fixed? > >> >> > >> > > >> > > >> > > >> > -- > >> > http://khangaonkar.blogspot.com/ > >> > > > > > > > > -- > > http://khangaonkar.blogspot.com/ >
