Re: [Spark Streaming] Measure latency

Hi Daniele, A pragmatic approach to do that would be to execute the computations in the scope of a foreachRDD, surrounded by wall-clock timers. For example: dstream.foreachRDD{ rdd => val t0 = System.currentTimeMillis() val aggregates = rdd. // make sure you get a result here, not

[Spark Streaming] Measure latency

Hi all, I am using spark streaming and I need to evaluate the latency of the standard aggregations (avg, min, max, …) provided by the spark APIs. Any way to do it in the code? Thanks in advance, --- Daniele