Kafka Consumer Behaviour

I was writing Kafka consumer and I have a query related to consumer processes. I have a consumer with groupId="testGroupId" and using the same groupId I consume from multiple topics say, "topic1" and "topic2". Also, assume "topic1" is already created on broker whereas "topic2" is not yet created.

Kafka Offset remains at -1

Dear Experts, I have a producer that pushes the Messages to kafka and a producer which is a spark algorithm to pick up the messages using a zookeeper instance. Below is the Spark log: Can anyone tell me what am I doing wrong? Kafka Topic details: *The command I used to create the Topic:* ./kaf

Re: Kafka Offset remains at -1

Are you sure your producer and consumer are pointed to the same topic? I see RECUR_TRANS in your logs but you mentioned creating RECURRING_TRANS. If thats not the issue. I have a few questions to get more informations. - Can you list & describe the relevant topics using the kafka-topics scr

Re: Partitions - Brokers - Servers

Hi, We normally run 1 broker per 1 physical server, and up to around 1000 partitions per broker (although that depends on the specific machine the broker is on and specific configuration). In order to enjoy replication, we recommend a minimum of 3 brokers in the cluster, to support 3 replicas per

Re: Kafka 0.9.0 release branch

A bit off topic but does this release contain the new single threaded consumer that supports the poll interface? Thanks! On Mon, Oct 12, 2015 at 1:31 PM, Jun Rao wrote: > Hi, Everyone, > > As we are getting closer to the 0.9.0 release, we plan to cut an 0.9.0 > release branch in about two weeks

Questions on the new Kafka consumer

I was reading the documentation for the new Kafka consumer API at https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java and came across this: "Each Kafka consumer must specify a consumer group that it belongs to." Currently we use Ka

Re: Questions on the new Kafka consumer

Aah I didn't read long enough. I just read the subscribing to specific partitions, managing your own offsets and the controlling the consumer's position sections. So a follow up question - when I am subscribing to specific partitions and managing my own offsets and controlling the consumer's posit

Default to port 9092

Newbies may see an odd message about array index out of bounds, if they neglect to specify the : in a kafka broker list entry. In the future, could Kafka be a little smarter about this, and just assume port 9092 when one isn't explicitly provided? Other network applications tend to work this way (c

Re: Kafka 0.9.0 release branch

Yes, 0.9 will include the new consumer. On Tue, Oct 13, 2015 at 12:50 PM, Rajiv Kurian wrote: > A bit off topic but does this release contain the new single threaded > consumer that supports the poll interface? > > Thanks! > > On Mon, Oct 12, 2015 at 1:31 PM, Jun Rao wrote: > > > Hi, Everyone,

r/apachekafka subreddit

Any Redditors in the house? I've created an r/apachekafka subreddit for anyone interested in Reddit-based discussions. https://www.reddit.com/r/apachekafka/ -- Cheers, Andrew

Re: Kafka 0.9.0 release branch

If time permits, it would be great to have both KAFKA-2397 and KAFKA-2017 in 0.9.0 as both of them are trying to fix some regressions of the new consumer compared with the old consumer. Priority-wise, I think 2017 is higher than 2397 since upon a broker shutdown / failure, all connected consumers

Re: Call to Future.get() hangs for producer

Jason, How did you validate that the message has been received at the server side? Guozhang On Fri, Oct 9, 2015 at 8:27 PM, Jason Kania wrote: > HI Guozhang, > Thanks for replying. I have minimal configuration at this point. I am only > trying to get the new producer working with a single kafk

partitions not syncing or becoming leaders

One of the three brokers in a Kafka cluster (v0.8.2.2) is seems be missing ISRs and even the partitions that have ISRs on it won’t be elected leaders for those partitions. Here’s a topic describe output: Topic:lia.stage.raw_events PartitionCount:20 ReplicationFactor:3 Configs:retention.ms=25920

Re: r/apachekafka subreddit

Subscribed :) Since the mailing list is rather active, I'm not sure there is significant benefit in a reddit community - but I'll be around to join discussions and see how it turns out. On Tue, Oct 13, 2015 at 1:58 PM, Andrew Pennebaker < andrew.penneba...@gmail.com> wrote: > Any Redditors in th