Hi,
What is the concurrency model behind SparkListener or StreamingListener
callbacks?
1. Multiple threads might access callbacks simultaneously.
2. Callbacks are guaranteed to be executed by a single thread.(Thread ids might
change on consecutive calls, though)
I asked the same question
StreamingListener {
private JavaStreamingContext jssc;
JobListener(JavaStreamingContext jssc) {
this.jssc = jssc;
}
@Override
public void
onBatchCompleted(StreamingListenerBatchCompleted
You should not call `jssc.stop(true);` in a StreamingListener. It will
cause a dead-lock: `jssc.stop` won't return until `listenerBus` exits. But
since `jssc.stop` blocks `StreamingListener`, `listenerBus` cannot exit.
Best Regards,
Shixiong Zhu
2015-06-04 0:39 GMT+08:00 dgoldenberg dgoldenberg
4, 2015 at 3:55 AM, Shixiong Zhu zsxw...@gmail.com wrote:
You should not call `jssc.stop(true);` in a StreamingListener. It will
cause a dead-lock: `jssc.stop` won't return until `listenerBus` exits. But
since `jssc.stop` blocks `StreamingListener`, `listenerBus` cannot exit.
Best Regards
JobListener is defined like so
private static class JobListener implements StreamingListener {
private JavaStreamingContext jssc;
JobListener(JavaStreamingContext jssc) {
this.jssc = jssc
At the end of foreachrdd i believe.
Thanks
Best Regards
On Thu, Mar 12, 2015 at 6:48 AM, Corey Nolet cjno...@gmail.com wrote:
Given the following scenario:
dstream.map(...).filter(...).window(...).foreachrdd()
When would the onBatchCompleted fire?
Given the following scenario:
dstream.map(...).filter(...).window(...).foreachrdd()
When would the onBatchCompleted fire?
Hi!
does anybody has some useful example of StreamingListener interface. When
and how can we use this interface to stop streaming when one batch of data
is processed?
Thanks alot
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Example-usage
Thanks Akhil
You are so helping Dear.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Example-usage-of-StreamingListener-tp20357p20362.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
