Vivek,
Did you say you have 8 spark jobs that are consuming from the same topic
and all jobs are using the same consumer group name? If so, each job would
get a subset of messages from that kafka topic, ie each job would get 1 out
of 8 messages from that topic. Is that your intent?
regards,
Duc
On Thu, Dec 24, 2015 at 7:20 AM, <vivek.meghanat...@wipro.com> wrote:
> We are using the older receiver based approach, the number of partitions
> is 1 (we have a single node kafka) and we use single thread per topic still
> we have the problem. Please see the API we use. All 8 spark jobs use same
> group name – is that a problem?
>
>
>
> val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap - *Number
> of threads used here is 1*
>
> val searches = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap).map(line => parse(line._2).extract[Search])
>
>
>
>
>
> Regards,
> Vivek M
>
> *From:* Bryan [mailto:bryan.jeff...@gmail.com]
> *Sent:* 24 December 2015 17:20
> *To:* Vivek Meghanathan (WT01 - NEP) <vivek.meghanat...@wipro.com>;
> user@spark.apache.org
> *Subject:* RE: Spark Streaming + Kafka + scala job message read issue
>
>
>
> Are you using a direct stream consumer, or the older receiver based
> consumer? If the latter, do the number of partitions you’ve specified for
> your topic match the number of partitions in the topic on Kafka?
>
>
>
> That would be an possible cause – as you might receive all data from a
> given partition while missing data from other partitions.
>
>
>
> Regards,
>
>
>
> Bryan Jeffrey
>
>
>
> Sent from Outlook Mail <http://go.microsoft.com/fwlink/?LinkId=550987>
> for Windows 10 phone
>
>
>
>
> *From: *vivek.meghanat...@wipro.com
> *Sent: *Thursday, December 24, 2015 5:22 AM
> *To: *user@spark.apache.org
> *Subject: *Spark Streaming + Kafka + scala job message read issue
>
>
>
> Hi All,
>
>
>
> We are using Bitnami Kafka 0.8.2 + spark 1.5.2 in Google cloud platform.
> Our spark streaming job(consumer) not receiving all the messages sent to
> the specific topic. It receives 1 out of ~50 messages(added log in the job
> stream and identified). We are not seeing any errors in the kafka logs.
> Unable to debug further from kafka layer. The console consumer shows the
> INPUT topic is received in the console. it is not reaching the spark-kafka
> integration stream. Any thoughts how to debug this issue. Another topic is
> working fine in same setup.
>
> Again tried with spark 1.3.0, kafka 0.8.1.1 which is also has same issue.
> All these jobs are working fine in our local lab servers
>
> Regards,
> Vivek M
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com
>
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com
>
