mridulm edited a comment on pull request #32790: URL: https://github.com/apache/spark/pull/32790#issuecomment-865364961
(Ignoring ephemenral nodes, dockerized deployments, etc - and using more real world scenarios) For a 5k cluster, the map can have upto 5k entries - with the values maps going of reasonable size (even if empty). For example, assuming 1k blocks per node on average - we end up with the map using approximately: ~ 5000 * 2048 * (~8 bytes) == ~80MB : this is the driver cost with zero blocks being used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
