Hey Bill! On 19 Jun 2015, at 16:24, Bill Sparks <[email protected]> wrote:
> Sorry for the post again. I guess I'm not understanding this… Thanks for posting again, not sorry! ;-) Regarding the broken link: where did you get this link? I think it should be referring here: http://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#parallel-execution For smaller machines, we usually suggest to set the number of slots per task manager to the number of cores, in your case 64. You will then have #taskmanager * #slotsPerTaskManager slots in your cluster, e.g. 1024. This should also be the parallelism you set for your program. Regarding configuration of network buffers: http://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#configuring-the-network-buffers With this high core count per machine, it is suggested to give the network layer quite some memory for concurrent shuffles. If you can afford it, go as high as a couple of GBs. The formula in the docs would suggest 262144 buffers, e.g. 8 GB. If this is not an option, I would decrease the number of slots per task manager (e.g. to 32) and reserve less buffers for the network stack. Feel free to post any time when something comes up. – Ufuk
