Hi
Broadcast variables definitely store in the spark.memory.storageFraction .
1 If we go into the code of TorrentBroadcast.scala and writeBlocks method
and navigates to BlockManager to MemoryStore . Desearlization of the
variables occures in unroll memory and then transferred to storage memory .
Satish,
I agree - that was my impression too. However I am seeing a smaller set of
storage memory used on a given executor than the amount of memory required for
my broadcast variables. I am wondering if the statistics in the ui are
incorrect or if the broadcasts are simply not a part of
My understanding is - it from storageFraction. Here cached blocks are
immune to eviction - so both persisted RDDs and broadcast variables sit
here. Ref
Hello.
Question: Do broadcast variables stored on executors count as part of
'storage memory' or other memory?
A little bit more detail:
I understand that we have two knobs to control memory allocation:
- spark.memory.fraction
- spark.memory.storageFraction
My understanding is that
