Re: Should the memory of worker nodes be constrained to the size of the master node?

Hi Darin, This is the piece of code doing the actual work (Setting the memory). As you can see, it leaves 15Gb of ram for OS on a > 100Gb machine... 2Gb RAM on a 10-20Gb machine etc. You can always set SPARK_WORKER_MEMORY/SPARK_EXECU

Should the memory of worker nodes be constrained to the size of the master node?

I started up a cluster on EC2 (using the provided scripts) and specified a different instance type for the master and the the worker nodes.  The cluster started fine, but when I looked at the cluster (via port 8080), it showed that the amount of memory available to the worker nodes did not match