Hi, I meet the same problem in the context of spark and yarn. When I open pyspark with the following command:
spark/bin/pyspark --master yarn-client --num-executors 1 --executor-memory 2500m It turns out *INFO storage.BlockManagerMasterActor: Registering block manager ip-10-0-6-171.us-west-2.compute.internal:38770 with 1294.1 MB RAM* So, according to the documentation, just 2156.83m is allocated to executor. Moreover, according to yarn 3072m memory is used for this container. Do you have any ideas about this? Thanks a lot Cheers Gen Boromir Widas wrote > Hey Larry, > > I have been trying to figure this out for standalone clusters as well. > http://apache-spark-user-list.1001560.n3.nabble.com/What-is-a-Block-Manager-td12833.html > has an answer as to what block manager is for. > > From the documentation, what I understood was if you assign X GB to each > executor, spark.storage.memoryFraction(default 0.6) * X is assigned to the > BlockManager and the rest for the JVM itself(?). > However, as you see, 26.8G is assigned to the BM, and assuming 0.6 > memoryFraction, this means the executor sees ~44.7G of memory, I am not > sure what happens to the difference(5.3G). > > > On Thu, Oct 9, 2014 at 9:40 PM, Larry Xiao < > xiaodi@.edu > > wrote: > >> Hi all, >> >> I'm confused about Executor and BlockManager, why they have different >> memory. >> >>> 14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added: >>> app-20141010085001-0000/2 on worker-20141010004933-brick6-35657 >>> (brick6:35657) with 6 cores >>> 14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor ID >>> app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores, 50.0 GB >>> RAM >>> >>> 14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block >>> manager >>> brick6:53296 with 26.8 GB RAM >>> >> and on the WebUI, >> >>> Executor ID Address RDD Blocks Memory Used Disk Used Active >>> Tasks Failed Tasks Complete Tasks Total Tasks Task Time >>> Input >>> Shuffle Read Shuffle Write >>> 0 brick3:37607 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 >>> 0 >>> ms 0.0 B 0.0 B 0.0 B >>> 1 brick1:59493 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 >>> 0 >>> ms 0.0 B 0.0 B 0.0 B >>> 2 brick6:53296 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 >>> 0 >>> ms 0.0 B 0.0 B 0.0 B >>> 3 brick5:38543 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 >>> 0 >>> ms 0.0 B 0.0 B 0.0 B >>> 4 brick2:44937 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 >>> 0 >>> ms 0.0 B 0.0 B 0.0 B >>> 5 brick4:46798 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 >>> 0 >>> ms 0.0 B 0.0 B 0.0 B >>> > <driver> > brick0:57692 0 0.0 B / 274.6 MB 0.0 B 0 0 0 >>> 0 0 ms 0.0 B 0.0 B 0.0 B >>> >> As I understand it, a worker consist of a daemon and an executor, and >> executor takes charge both execution and storage. >> So does it mean that 26.8 GB is saved for storage and the rest is for >> execution? >> >> Another question is that, throughout execution, it seems that the >> blockmanager is always almost free. >> >>> 14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in >>> memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB) >>> >> I don't know what I'm missing here. >> >> Best regards, >> Larry >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: > user-unsubscribe@.apache >> For additional commands, e-mail: > user-help@.apache >> >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Executor-and-BlockManager-memory-size-tp16091p17816.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
