Re: Stopping StreamingContext does not kill receiver
I'm running on spark 1.0.0 and I see a similar problem when using the socketTextStream receiver. The ReceiverTracker task sticks around after a ssc.stop(false). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Stopping StreamingContext does not kill receiver
Can you give the stack trace? This was the fix for the twitter stream. https://github.com/apache/spark/pull/1577/files You could try doing the same. TD On Wed, Aug 6, 2014 at 2:41 PM, lbustelo g...@bustelos.com wrote: I'm running on spark 1.0.0 and I see a similar problem when using the socketTextStream receiver. The ReceiverTracker task sticks around after a ssc.stop(false). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Stopping StreamingContext does not kill receiver
I narrowed down the error. Unfortunately this is not quick fix. I have opened a JIRA for this. https://issues.apache.org/jira/browse/SPARK-2892 On Wed, Aug 6, 2014 at 3:59 PM, Tathagata Das tathagata.das1...@gmail.com wrote: Okay let me give it a shot. On Wed, Aug 6, 2014 at 3:57 PM, lbustelo g...@bustelos.com wrote: Sorry about the screenshot… but that is what I have handy right now. You can see that we get a WARN and it ultimately say that it stopped successfully. When looking that the application in Spark UI, it still shows the ReceiverTracker task running. It is easy to recreate. On the spark repl we are running a modified version of https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala . Then do a ssc.stop(false). http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Stopping StreamingContext does not kill receiver
From the interactive shell I’ve created a StreamingContext. I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active. Is this expected? I’m running 1.0.1 on EC2. Nick -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Stopping StreamingContext does not kill receiver
Yes, thats a bug i just discovered. Race condition in the Twitter Receiver, will fix asap. Here is the JIRA https://issues.apache.org/jira/browse/SPARK-2464 TD On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: To add a potentially relevant piece of information, around when I stop the StreamingContext, I get the following warning: 14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 - ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-140520314#-273455949],true,url-here-too,,)) Nick On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas nicholas.cham...@gmail.com wrote: From the interactive shell I’ve created a StreamingContext. I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active. Is this expected? I’m running 1.0.1 on EC2. Nick -- View this message in context: Stopping StreamingContext does not kill receiver http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Stopping StreamingContext does not kill receiver
Okie doke. Thanks for filing the JIRA. On Sat, Jul 12, 2014 at 6:45 PM, Tathagata Das tathagata.das1...@gmail.com wrote: Yes, thats a bug i just discovered. Race condition in the Twitter Receiver, will fix asap. Here is the JIRA https://issues.apache.org/jira/browse/SPARK-2464 TD On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: To add a potentially relevant piece of information, around when I stop the StreamingContext, I get the following warning: 14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 - ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-140520314#-273455949],true,url-here-too,,)) Nick On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas nicholas.cham...@gmail.com wrote: From the interactive shell I’ve created a StreamingContext. I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active. Is this expected? I’m running 1.0.1 on EC2. Nick -- View this message in context: Stopping StreamingContext does not kill receiver http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
