Master and Worker are components of the Spark's standalone cluster manager, which manages the available resources in a cluster and divides them between different Spark applications. A spark application's Driver asks the Master for resources. Master allocates certain Workers to the application. Those Worker start running Executors to do processing for that application. The Driver then launches tasks directly to those Executors.
This is pictographically explained here http://spark.incubator.apache.org/docs/latest/cluster-overview.html . Maybe this will help explain it better. TD On Fri, Jan 24, 2014 at 9:59 AM, Manoj Samel <manojsamelt...@gmail.com>wrote: > On cluster with HDFS + Spark (in standalone deploy mode), there is a > master node + 4 worker nodes. When a spark-shell connects to master, it > creates 4 executor JVMs on each of the 4 worker nodes. > > When the application reads a HDFS files and does computations in RDDs, > what work gets done on master, worker, executor and driver ? > > Thanks, >
