Re: Kafka directsream receiving rate

2016-02-08 Thread Diwakar Dhanuskodi
Samsung Mobile. Original message From: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> Date:07/02/2016 01:39 (GMT+05:30) To: Cody Koeninger <c...@koeninger.org> Cc: user@spark.apache.org Subject: Re: Kafka directsream receiving rate Thanks Cody for trying to

Re: Kafka directsream receiving rate

2016-02-06 Thread Diwakar Dhanuskodi
org> Date:06/02/2016 22:32 (GMT+05:30) To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> Cc: user@spark.apache.org Subject: Re: Kafka directsream receiving rate I am not at all clear on what you are saying. "Yes , I am printing each messages . It is processing all messages u

Re: Kafka directsream receiving rate

2016-02-06 Thread Cody Koeninger
er batch . > > > > > Sent from Samsung Mobile. > > > Original message > From: Cody Koeninger <c...@koeninger.org> > Date:06/02/2016 01:30 (GMT+05:30) > To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> > Cc: user@spark.apache.org >

Re: Kafka directsream receiving rate

2016-02-06 Thread Diwakar Dhanuskodi
Mobile. Original message From: Cody Koeninger <c...@koeninger.org> Date:06/02/2016 01:30 (GMT+05:30) To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> Cc: user@spark.apache.org Subject: Re: Kafka directsream receiving rate Have you tried just printing each message, to see w

Re: Kafka directsream receiving rate

2016-02-05 Thread Cody Koeninger
If you're using the direct stream, you have 0 receivers. Do you mean you have 1 executor? Can you post the relevant call to createDirectStream from your code, as well as any relevant spark configuration? On Thu, Feb 4, 2016 at 8:13 PM, Diwakar Dhanuskodi < diwakar.dhanusk...@gmail.com> wrote:

Re: Kafka directsream receiving rate

2016-02-05 Thread Cody Koeninger
> /root/dime/jars/spark-streaming-kafka-assembly_2.10-1.5.1.jar,/root/Jars/sparkreceiver.jar > /root/Jars/sparkreceiver.jar > > > > > Sent from Samsung Mobile. > > > Original message > From: Cody Koeninger <c...@koeninger.org> > Date:05/02/2016 22:07 (

Re: Kafka directsream receiving rate

2016-02-05 Thread Diwakar Dhanuskodi
er.jar /root/Jars/sparkreceiver.jar Sent from Samsung Mobile. Original message From: Cody Koeninger <c...@koeninger.org> Date:05/02/2016 22:07 (GMT+05:30) To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> Cc: user@spark.apache.org Subject: Re: Kaf

Re: Kafka directsream receiving rate

2016-02-05 Thread Diwakar Dhanuskodi
<c...@koeninger.org> Date:06/02/2016 00:33 (GMT+05:30) To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> Cc: user@spark.apache.org Subject: Re: Kafka directsream receiving rate How are you counting the number of messages? I'd go ahead and remove the settings for

Re: Kafka directsream receiving rate

2016-02-05 Thread Cody Koeninger
:33 (GMT+05:30) > To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> > Cc: user@spark.apache.org > Subject: Re: Kafka directsream receiving rate > > How are you counting the number of messages? > > I'd go ahead and remove the settings for backpressure and > maxrateperpartit

RE: Kafka directsream receiving rate

2016-02-04 Thread Diwakar Dhanuskodi
Adding more info Batch  interval  is  2000ms. I expect all 100 messages  go thru one  dstream from  directsream but it receives at rate of 10 messages at time. Am  I missing  some  configurations here. Any help appreciated.  Regards  Diwakar. Sent from Samsung Mobile. Original