On Mon, Apr 27, 2015 at 7:36 AM, Shuai Zheng <szheng.c...@gmail.com> wrote:
> Thanks. So may I know what is your configuration for more/smaller > executors on r3.8xlarge, how big of the memory that you eventually decide > to give one executor without impact performance (for example: 64g? ). > We're currently using 16 executors with 2 cores each per instance. This is mainly due to the S3 throughput observations I mentioned (all based on earlier versions, so I would not be surprised if larger executors will work equally well now). Among these, we chop the available memory up equally, but we still leave a lot of extra room for Python workers (as a result of SPARK-5395; since that's fixed, we have more room to grow the JVM heap now). Best, -Sven -- www.skrasser.com <http://www.skrasser.com/?utm_source=sig>
