Now I see the answer to this. Spark slaves are start on random ports, and tell the master where they are. then the master acknowledges them.
(worker logs) Starting Spark worker xxxx:43282 (master logs) Registering worker on xxxx:43282 with 8 cores, 16.5 GB RAM Thus, the port is random because the slaves can be ephemeral. Since the master is fixed, though, a new slave can reconnect at any time. On Mon, Jul 14, 2014 at 10:01 PM, jay vyas <jayunit100.apa...@gmail.com> wrote: > Hi spark ! > > What is the purpose of the randomly assigned SPARK_WORKER_PORT > > from the documentation it sais to "join a cluster", but its not clear to > me how a random port could be used to communicate with other members of a > spark pool. > > This question might be grounded in my ignorance ... if so please just > point me to the right documentation if im mising something obvious :) > > thanks ! > -- > jay vyas > -- jay vyas
