[
https://issues.apache.org/jira/browse/SPARK-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240437#comment-14240437
]
Ilayaperumal Gopinathan commented on SPARK-2892:
------------------------------------------------
It looks like this one and the issue mentioned in SPARK-4802 (ReceiverInfo
removal at ReceiverTracker upon deregistering receiver) are related. I believe
the following warning message is the result of receiverInfo not being removed
at ReceiverTracker by the ReceiverTrackerActor when the corresponding receiver
is deregistered.
"WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 ->
ReceiverInfo(0,SocketReceiver-0,null,false,localhost,Stopped by driver,))"
>From what I can see so far, closing the streaming context stops the receiver
>only in "local" mode.
In "cluster" mode, using the Spark standalone cluster I noticed that when the
ReceiverTracker at the driver sends the "StopReceiver" message as a result of
streaming context close, it couldn't reach to the ReceiverSupervisorImpl's
actor that is running at the executor node. At the same time, the
ReceiverSupervisorImpl at the executor could send the messages such as
RegisterReceiver, AddBlock back to the ReceiverTrackerActor at the driver.
It would be great if someone could explain what might be going on from
ReceiverTracker -> ReceiverSupervisorImpl actor at executor when sending the
stop signal in the distributed mode case.
Thanks!
> Socket Receiver does not stop when streaming context is stopped
> ---------------------------------------------------------------
>
> Key: SPARK-2892
> URL: https://issues.apache.org/jira/browse/SPARK-2892
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.0.2
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Priority: Critical
>
> Running NetworkWordCount with
> {quote}
> ssc.start(); Thread.sleep(10000); ssc.stop(stopSparkContext = false);
> Thread.sleep(60000)
> {quote}
> gives the following error
> {quote}
> 14/08/06 18:37:13 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0)
> in 10047 ms on localhost (1/1)
> 14/08/06 18:37:13 INFO DAGScheduler: Stage 0 (runJob at
> ReceiverTracker.scala:275) finished in 10.056 s
> 14/08/06 18:37:13 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> 14/08/06 18:37:13 INFO SparkContext: Job finished: runJob at
> ReceiverTracker.scala:275, took 10.179263 s
> 14/08/06 18:37:13 INFO ReceiverTracker: All of the receivers have been
> terminated
> 14/08/06 18:37:13 WARN ReceiverTracker: All of the receivers have not
> deregistered, Map(0 ->
> ReceiverInfo(0,SocketReceiver-0,null,false,localhost,Stopped by driver,))
> 14/08/06 18:37:13 INFO ReceiverTracker: ReceiverTracker stopped
> 14/08/06 18:37:13 INFO JobGenerator: Stopping JobGenerator immediately
> 14/08/06 18:37:13 INFO RecurringTimer: Stopped timer for JobGenerator after
> time 1407375433000
> 14/08/06 18:37:13 INFO JobGenerator: Stopped JobGenerator
> 14/08/06 18:37:13 INFO JobScheduler: Stopped JobScheduler
> 14/08/06 18:37:13 INFO StreamingContext: StreamingContext stopped successfully
> 14/08/06 18:37:43 INFO SocketReceiver: Stopped receiving
> 14/08/06 18:37:43 INFO SocketReceiver: Closed socket to localhost:9999
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
