Hi, I am planning to run spark on EMR. And because my application might take a lot of memory. On EMR, I know there is a hard limit 16G physical memory on individual mapper/reducer (otherwise I will have an exception and this is confirmed by AWS EMR team, at least it is the spec at this moment).
And if I use Yarn on EMR, and submit the spark job to YARN, I assume the yarn will take the responsibility to do the resource allocation, so the limitation on the physical memory still be 16G? Is it a reasonable guess or anyone has any experience to use more than 16G memory on the EMR for individual executor? And I notice that there are some examples that allocate more than 16G memory in the doc, so if I use spark cluster by itself, I can use more memory? Regards, Shuai
