In yarn-client mode, the driver is on the machine where you ran the spark-submit. The executors are running in the YARN cluster nodes, and the socket receiver listening on port 9999 is running in one of the executors.
On Mon, Aug 10, 2015 at 11:43 AM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > I am running as a yarn-client which probably means that the program that > submitted the job is where the listening is also occurring? I thought that > the yarn is only used to negotiate resources in yarn-client master mode. > > On Mon, Aug 10, 2015 at 11:34 AM, Tathagata Das <t...@databricks.com> > wrote: > >> If you are running on a cluster, the listening is occurring on one of the >> executors, not in the driver. >> >> On Mon, Aug 10, 2015 at 10:29 AM, Mohit Anchlia <mohitanch...@gmail.com> >> wrote: >> >>> I am trying to run this program as a yarn-client. The job seems to be >>> submitting successfully however I don't see any process listening on this >>> host on port 9999 >>> >>> >>> https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java >>> Active Jobs (2)Job IdDescriptionSubmittedDurationStages: >>> Succeeded/TotalTasks >>> (for all stages): Succeeded/Total1foreachRDD at >>> JavaRecoverableNetworkWordCount.java:112 >>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=1>2015/08/10 >>> 13:27:3651 s0/2 >>> 0/2 >>> 0start at JavaRecoverableNetworkWordCount.java:152 >>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=0>2015/08/10 >>> 13:27:3551 s0/2 >>> 0/70 >>> >> >> >
