,
Raymond Liu
From: Victor Tso-Guillen [mailto:v...@paxata.com]
Sent: Wednesday, August 27, 2014 1:40 PM
To: Liu, Raymond
Cc: user@spark.apache.org
Subject: Re: What is a Block Manager?
We're a single-app deployment so we want to launch as many executors as the
system has workers. We accomplish
[mailto:v...@paxata.com]
Sent: Wednesday, August 27, 2014 1:40 PM
To: Liu, Raymond
Cc: user@spark.apache.org
Subject: Re: What is a Block Manager?
We're a single-app deployment so we want to launch as many executors as
the system has workers. We accomplish this by not configuring the max
Basically, a Block Manager manages the storage for most of the data in spark,
name a few: block that represent a cached RDD partition, intermediate shuffle
data, broadcast data etc. it is per executor, while in standalone mode,
normally, you have one executor per worker.
You don't control how
We're a single-app deployment so we want to launch as many executors as the
system has workers. We accomplish this by not configuring the max for the
application. However, is there really no way to inspect what
machines/executor ids/number of workers/etc is available in context? I'd
imagine that
