Re: abnormal latency when running Spark Streaming

[Boduo Li]

Hi Yingjun,

Do you see a stable latency or the latency keeps increasing? And could you
provide some details about the input data rate/node, batch interval,
windowDuration and slideDuration when you see the high latency?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/abnormal-latency-when-running-Spark-Streaming-tp7315p7324.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

abnormal latency when running Spark Streaming

[Yingjun Wu]

);

wordCounts.count().map(new Function() {
private long startTime = System.currentTimeMillis();

@Override
public Long call(Long in) throws Exception {
long endTime = System.currentTimeMillis();
String mystring = 
"=output===\nstartTime="
+ startTime
+ ", endTime="
+ endTime
+ ", elapsedTime="
+ (endTime-startTime)*1.0/1000
+ 
"sec\n==\n";
System.out.println(mystring);
return in;
}
}).print();
jssc.start();
jssc.awaitTermination();
}
}

I am wondering is there anything wrong with my source code? Or is my method
to measure the latency problematic? Thanks.

Regards,
Yingjun




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/abnormal-latency-when-running-Spark-Streaming-tp7315.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.