Hey Larry, I have been trying to figure this out for standalone clusters as well. http://apache-spark-user-list.1001560.n3.nabble.com/What-is-a-Block-Manager-td12833.html has an answer as to what block manager is for.
>From the documentation, what I understood was if you assign X GB to each executor, spark.storage.memoryFraction(default 0.6) * X is assigned to the BlockManager and the rest for the JVM itself(?). However, as you see, 26.8G is assigned to the BM, and assuming 0.6 memoryFraction, this means the executor sees ~44.7G of memory, I am not sure what happens to the difference(5.3G). On Thu, Oct 9, 2014 at 9:40 PM, Larry Xiao <xia...@sjtu.edu.cn> wrote: > Hi all, > > I'm confused about Executor and BlockManager, why they have different > memory. > >> 14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added: >> app-20141010085001-0000/2 on worker-20141010004933-brick6-35657 >> (brick6:35657) with 6 cores >> 14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor ID >> app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores, 50.0 GB RAM >> >> 14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block manager >> brick6:53296 with 26.8 GB RAM >> > and on the WebUI, > >> Executor ID Address RDD Blocks Memory Used Disk Used Active >> Tasks Failed Tasks Complete Tasks Total Tasks Task Time Input >> Shuffle Read Shuffle Write >> 0 brick3:37607 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0 >> ms 0.0 B 0.0 B 0.0 B >> 1 brick1:59493 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0 >> ms 0.0 B 0.0 B 0.0 B >> 2 brick6:53296 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0 >> ms 0.0 B 0.0 B 0.0 B >> 3 brick5:38543 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0 >> ms 0.0 B 0.0 B 0.0 B >> 4 brick2:44937 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0 >> ms 0.0 B 0.0 B 0.0 B >> 5 brick4:46798 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0 >> ms 0.0 B 0.0 B 0.0 B >> <driver> brick0:57692 0 0.0 B / 274.6 MB 0.0 B 0 0 0 >> 0 0 ms 0.0 B 0.0 B 0.0 B >> > As I understand it, a worker consist of a daemon and an executor, and > executor takes charge both execution and storage. > So does it mean that 26.8 GB is saved for storage and the rest is for > execution? > > Another question is that, throughout execution, it seems that the > blockmanager is always almost free. > >> 14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in >> memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB) >> > I don't know what I'm missing here. > > Best regards, > Larry > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
