consumer (java client 2.11 0.9.0.1) poll does not return record in between
I am using 0.9.0.1 JAVA KafkaCosnumer poll API with timeout as long as 1000ms. I am connecting to a topic with 7 million messages. The message is FIX message. When I poll - I see that in between it returns blank records. Why is that happening. Is there a possibility that poll API is returning the same offset from a specific partition more than once. I have two partitions on the topic and replication factor is 2 with retention set to 4 years on this topic. The cluster has three brokers and thee ZK nodes. -- *Warm Regards,* *Shashank * *Mobile: +91 9910478553 * *Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/ <http://in.linkedin.com/pub/shashank-singh/13/763/906/>*
new consumer API & release 0.8.3
Hi I am eager to get to use the enhanced Consumer API which provides better control in terms of offset management etc. As I believe from reading through forums it is coming as part of 0.8.3 release. However there is no tentative date for the same. Can you please give any hint on that. Also which is the best forum to ask questions on how these new APIs are shaping up and the details about the same.. -- *Warm Regards,* *Shashank * *Mobile: +91 9910478553 * *Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/ <http://in.linkedin.com/pub/shashank-singh/13/763/906/>*
kafka consumer group API
Hi Team I was going over the documentation and links related to consumer API provided in 0.8.2.1 release I got to know of major improvement in consumer API in 0.8.3 release. May I know when we are planning to have this release. One more help - I was following up the example given on https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example In this approach we can't decide a) how to get the current offset b) how to set a particular consumer group to start reading from a specific offset Also how to use the same example to read one message at a time from a topic with many topics. I tried with creating a list of iterators and then checking for hasNext() but that call in blocking. Even though I am using a timeout on that call this seems not be working. Instead what I have settled with is an internal queue to which the multi-threaded consumer offers the messages into. Poll API polls the internal queue. Problem is in case client code gets down without consuming all the messages the offset is already set and the messages which have already been read from iterator and put in queue will be never seen again. -- *Warm Regards,* *Shashank * *Mobile: +91 9910478553 * *Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/ <http://in.linkedin.com/pub/shashank-singh/13/763/906/>*
