I would check out the source examples on Spark's Github:
https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples
And, Zhen He put together a great web page with summaries and examples of
each function:
Long story [1] short, akka opens up dynamic, random ports for each job [2].
So, simple NAT fails. You might try some trickery with a DNS server and
docker's --net=host .
[1]
http://apache-spark-user-list.1001560.n3.nabble.com/Comprehensive-Port-Configuration-reference-tt5384.html#none
[2]
@spark.apache.org
Date: 05/28/2014 05:18 PM
Subject:Re: Comprehensive Port Configuration reference?
Hmm, those do look like 4 listening ports to me. PID 3404 is an executor
and PID 4762 is a worker? This is a standalone cluster?
On Wed, May 28, 2014 at 8:22 AM, Jacob Eisinger jeis
-to-send-to
Thanks for taking a look through!
I also realized that I had a couple mistakes with the 0.9 to 1.0 transition
so appropriately documented those now as well in the updated PR.
Cheers!
Andrew
On Fri, May 23, 2014 at 2:43 PM, Jacob Eisinger jeis...@us.ibm.com wrote:
Howdy Andrew,
I
.
On Tue, May 6, 2014 at 3:30 PM, Jacob Eisinger jeis...@us.ibm.com wrote:
Howdy,
You might find the discussion Andrew and I have been having about Docker
and network security [1] applicable.
Also, I posted an answer [2] to your stackoverflow question.
[1]
http://apache-spark-user-list
Howdy,
You might find the discussion Andrew and I have been having about Docker
and network security [1] applicable.
Also, I posted an answer [2] to your stackoverflow question.
[1]
Howdy Scott,
Please see the discussions about securing the Spark network [1] [2].
In a nut shell, Spark opens up a couple of well known ports. And,then the
workers and the shell open up dynamic ports for each job. These dynamic
ports make securing the Spark network difficult.
Jacob
[1]
Howdy Andrew,
I think I am running into the same issue [1] as you. It appears that Spark
opens up dynamic / ephemera [2] ports for each job on the shell and the
workers. As you are finding out, this makes securing and managing the
network for Spark very difficult.
Any idea how to restrict
being used
https://groups.google.com/forum/#!topic/spark-users/PN0WoJiB0TA
On Fri, Apr 25, 2014 at 8:53 PM, Jacob Eisinger jeis...@us.ibm.com wrote:
Howdy,
We tried running Spark 0.9.1 stand-alone inside docker containers
distributed over multiple hosts. This is complicated due to Spark
