This should give accurate count for each batch, though for getting the rate
you have to make sure that you streaming app is stable, that is, batches
are processed as fast as they are received (scheduling delay in the spark
streaming UI is approx 0).
TD
On Tue, Jun 23, 2015 at 2:49 AM, anshu shukla <anshushuk...@gmail.com>
wrote:
> I am calculating input rate using the following logic.
>
> And i think this foreachRDD is always running on driver (println are seen on
> driver)
>
> 1- Is there any other way to do that in less cost .
>
> 2- Will this give me the correct count for rate .
>
>
> //code -
>
> inputStream.foreachRDD(new Function<JavaRDD<String>, Void>() {
> @Override
> public Void call(JavaRDD<String> stringJavaRDD) throws Exception {
> System.out.println(System.currentTimeMillis()+",spoutstringJavaRDD,"
> + stringJavaRDD.count() );
> return null;
> }
> });
>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
