I have a large Map that is assembled in the driver and broadcast to each node.
My question is how best to allocate memory for this. The Driver has to have enough memory for the Maps, but only one copy is serialized to each node. What type of memory should I size to match the Maps? Is the broadcast Map taking a little from each executor, all from every executor, or is there something other than driver and executor memory I can size? --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
