I would like to get a sense of spark YARN cluster used around and this thread can help others as well
1. Number of nodes in cluster 2. Container memory limit 3. Typical Hardware configuration of worker nodes 4. Typical number of executors used ? 5. Any other related info you want to share. How do you decide on number of executors/cores/memory given you know the amount of data you will process with/without cache enabled. -- Deepak
