Re: Stopping StreamingContext does not kill receiver

2014-08-06 Thread Tathagata Das 
I narrowed down the error. Unfortunately this is not quick fix. I have
opened a JIRA for this.
https://issues.apache.org/jira/browse/SPARK-2892


On Wed, Aug 6, 2014 at 3:59 PM, Tathagata Das 
wrote:

> Okay let me give it a shot.
>
>
> On Wed, Aug 6, 2014 at 3:57 PM, lbustelo  wrote:
>
>> Sorry about the screenshot… but that is what I have handy right now. You
>> can
>> see that we get a WARN and it ultimately say that it stopped successfully.
>> When looking that the application in Spark UI, it still shows the
>> ReceiverTracker task running.
>>
>> It is easy to recreate. On the spark repl we are running a modified
>> version
>> of
>>
>> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala
>> .
>>
>> Then do a ssc.stop(false).
>>
>> <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png
>> >
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Re: Stopping StreamingContext does not kill receiver

2014-08-06 Thread Tathagata Das 
Okay let me give it a shot.


On Wed, Aug 6, 2014 at 3:57 PM, lbustelo  wrote:

> Sorry about the screenshot… but that is what I have handy right now. You
> can
> see that we get a WARN and it ultimately say that it stopped successfully.
> When looking that the application in Spark UI, it still shows the
> ReceiverTracker task running.
>
> It is easy to recreate. On the spark repl we are running a modified version
> of
>
> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala
> .
>
> Then do a ssc.stop(false).
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png
> >
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Stopping StreamingContext does not kill receiver

2014-08-06 Thread lbustelo 
Sorry about the screenshot… but that is what I have handy right now. You can
see that we get a WARN and it ultimately say that it stopped successfully.
When looking that the application in Spark UI, it still shows the
ReceiverTracker task running.

It is easy to recreate. On the spark repl we are running a modified version
of
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala.

Then do a ssc.stop(false).


 



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Stopping StreamingContext does not kill receiver

2014-08-06 Thread Tathagata Das 
Can you give the stack trace?

This was the fix for the twitter stream.

https://github.com/apache/spark/pull/1577/files

You could try doing the same.

TD



On Wed, Aug 6, 2014 at 2:41 PM, lbustelo  wrote:

> I'm running on spark 1.0.0 and I see a similar problem when using the
> socketTextStream receiver. The ReceiverTracker task sticks around after a
> ssc.stop(false).
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Stopping StreamingContext does not kill receiver

2014-08-06 Thread lbustelo 
I'm running on spark 1.0.0 and I see a similar problem when using the
socketTextStream receiver. The ReceiverTracker task sticks around after a
ssc.stop(false).



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Stopping StreamingContext does not kill receiver

2014-07-12 Thread Nicholas Chammas 
Okie doke. Thanks for filing the JIRA.


On Sat, Jul 12, 2014 at 6:45 PM, Tathagata Das 
wrote:

> Yes, thats a bug i just discovered. Race condition in the Twitter
> Receiver, will fix asap.
> Here is the JIRA https://issues.apache.org/jira/browse/SPARK-2464
>
> TD
>
>
> On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas <
> nicholas.cham...@gmail.com> wrote:
>
>> To add a potentially relevant piece of information, around when I stop
>> the StreamingContext, I get the following warning:
>>
>> 14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not 
>> deregistered, Map(0 -> 
>> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-140520314#-273455949],true,url-here-too,,))
>>
>> Nick
>> ​
>>
>>
>> On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas > > wrote:
>>
>>> From the interactive shell I’ve created a StreamingContext.
>>>
>>> I call ssc.start() and take a look at http://master_url:4040/streaming/
>>> and see that I have an active Twitter receiver. Then I call 
>>> ssc.stop(stopSparkContext
>>> = false, stopGracefully = true) and wait a bit, but the receiver seems
>>> to stay active.
>>>
>>> Is this expected? I’m running 1.0.1 on EC2.
>>>
>>> Nick
>>> ​
>>>
>>> --
>>> View this message in context: Stopping StreamingContext does not kill
>>> receiver
>>> 
>>> Sent from the Apache Spark User List mailing list archive
>>>  at Nabble.com.
>>>
>>
>>
>

Re: Stopping StreamingContext does not kill receiver

2014-07-12 Thread Tathagata Das 
Yes, thats a bug i just discovered. Race condition in the Twitter Receiver,
will fix asap.
Here is the JIRA https://issues.apache.org/jira/browse/SPARK-2464

TD


On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:

> To add a potentially relevant piece of information, around when I stop the
> StreamingContext, I get the following warning:
>
> 14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not 
> deregistered, Map(0 -> 
> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-140520314#-273455949],true,url-here-too,,))
>
> Nick
> ​
>
>
> On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas 
> wrote:
>
>> From the interactive shell I’ve created a StreamingContext.
>>
>> I call ssc.start() and take a look at http://master_url:4040/streaming/
>> and see that I have an active Twitter receiver. Then I call 
>> ssc.stop(stopSparkContext
>> = false, stopGracefully = true) and wait a bit, but the receiver seems
>> to stay active.
>>
>> Is this expected? I’m running 1.0.1 on EC2.
>>
>> Nick
>> ​
>>
>> --
>> View this message in context: Stopping StreamingContext does not kill
>> receiver
>> 
>> Sent from the Apache Spark User List mailing list archive
>>  at Nabble.com.
>>
>
>

Re: Stopping StreamingContext does not kill receiver

2014-07-12 Thread Nicholas Chammas 
To add a potentially relevant piece of information, around when I stop the
StreamingContext, I get the following warning:

14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not
deregistered, Map(0 ->
ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-140520314#-273455949],true,url-here-too,,))

Nick
​


On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas 
wrote:

> From the interactive shell I’ve created a StreamingContext.
>
> I call ssc.start() and take a look at http://master_url:4040/streaming/
> and see that I have an active Twitter receiver. Then I call 
> ssc.stop(stopSparkContext
> = false, stopGracefully = true) and wait a bit, but the receiver seems to
> stay active.
>
> Is this expected? I’m running 1.0.1 on EC2.
>
> Nick
> ​
>
> --
> View this message in context: Stopping StreamingContext does not kill
> receiver
> 
> Sent from the Apache Spark User List mailing list archive
>  at Nabble.com.
>