Re: Calculating tuple count /input rate with time

2015-06-23 Thread Tathagata Das 
This should give accurate count for each batch, though for getting the rate
you have to make sure that you streaming app is stable, that is, batches
are processed as fast as they are received (scheduling delay in the spark
streaming UI is approx 0).

TD

On Tue, Jun 23, 2015 at 2:49 AM, anshu shukla anshushuk...@gmail.com
wrote:

 I am calculating input rate using the following logic.

 And i think this foreachRDD is always running on driver (println are seen on 
 driver)

 1- Is there any other way to do that in less cost .

 2- Will this give me the correct count for rate  .


 //code -

 inputStream.foreachRDD(new FunctionJavaRDDString, Void() {
 @Override
 public Void call(JavaRDDString stringJavaRDD) throws Exception {
 System.out.println(System.currentTimeMillis()+,spoutstringJavaRDD, 
 + stringJavaRDD.count() );
 return null;
 }
 });



 --
 Thanks  Regards,
 Anshu Shukla

Calculating tuple count /input rate with time

2015-06-23 Thread anshu shukla 
I am calculating input rate using the following logic.

And i think this foreachRDD is always running on driver (println are
seen on driver)

1- Is there any other way to do that in less cost .

2- Will this give me the correct count for rate  .


//code -

inputStream.foreachRDD(new FunctionJavaRDDString, Void() {
@Override
public Void call(JavaRDDString stringJavaRDD) throws Exception {
System.out.println(System.currentTimeMillis()+,spoutstringJavaRDD,
+ stringJavaRDD.count() );
return null;
}
});



-- 
Thanks  Regards,
Anshu Shukla