I think I have this right:
You will run one executor per application per worker. Generally there
is one worker per machine, and it manages all of the machine's
resources. So if you want one app to use this whole machine you need
to ask for 48G and 24 cores. That's better than splitting up the
Hi, I have some questions about how the spark run the job concurrently.
For example, if I setup the Spark on one standalone test box, which has 24 core
and 64G memory. I setup the Worker memory to 48G, and Executor memory to 4G,
and using spark-shell to run some jobs. Here is something confusing
On Sun, Feb 8, 2015 at 10:26 PM, java8964 java8...@hotmail.com wrote:
standalone one box environment, if I want to use all 48G memory allocated to
worker for my application, I should ask 48G memory for the executor in the
spark shell, right? Because 48G is too big for a JVM heap in normal case,
