To add some internals, the high level consumer actually does read entire batches from Kafka. It just exposes them to the user in an event loop, because its a very natural API. Users can then batch events the way they prefer.
So if you are worried about batches being more efficient than single events, you are covered! Gwen On Tue, Aug 4, 2015 at 12:04 PM, shahab <shahab.mok...@gmail.com> wrote: > Thanks a lot Shaminder for clarification and thanks Raja for pointing me to > the example. > > best, > /shahab > > On Tue, Aug 4, 2015 at 6:06 PM, Rajasekar Elango <rela...@salesforce.com> > wrote: > > > Here is an example on what sharninder suggested > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > > Thanks, > > Raja. > > > > On Tue, Aug 4, 2015 at 12:01 PM, Sharninder <sharnin...@gmail.com> > wrote: > > > > > You can't. Kafka is essentially a queue, so you always read messages > one > > > by one. What you can do is disable auto offset commit, read 100 > messages, > > > process them and then manually commit offset. > > > > > > -- > > > Sharninder > > > > > > > On 04-Aug-2015, at 9:07 pm, shahab <shahab.mok...@gmail.com> wrote: > > > > > > > > Hi, > > > > > > > > While we the producer can put data as batch in kafka server, I > > couldn't > > > > find any API (or any document) saying how we can fetch data as batch > > from > > > > Kafka ? > > > > Even when data is placed as batch in kafka server, still using High > > Level > > > > consumer I can only read one by one, and I can not specify. for > > example, > > > > read 100 items at once! > > > > > > > > Is this correct observation? or I am missing something? > > > > > > > > best, > > > > /Shahab > > > > > > > > > > > -- > > Thanks, > > Raja. > > >
