Hello. Question: Do broadcast variables stored on executors count as part of 'storage memory' or other memory?
A little bit more detail: I understand that we have two knobs to control memory allocation: - spark.memory.fraction - spark.memory.storageFraction My understanding is that spark.memory.storageFraction controls the amount of memory allocated for cached RDDs. spark.memory.fraction controls how much memory is allocated to Spark operations (task serialization, operations, etc.), w/ the remainder reserved for user data structures, Spark internal metadata, etc. This includes the storage memory for cached RDDs. You end up with executor memory that looks like the following: All memory: 0-100 Spark memory: 0-75 RDD Storage: 0-37 Other Spark: 38-75 Other Reserved: 76-100 Where do broadcast variables fall into the mix? Regards, Bryan Jeffrey
