It makes sense. Thank you very much, Jamie!
> On Jan 15, 2019, at 12:48 PM, Jamie Grier <jgr...@lyft.com> wrote: > > Ethan, it depends on what you mean by easy ;) It just depends a lot on what > infra tools you already have in place. On bare metal it's probably safe to > say there is no "easy" way. You need a lot of automation to make it easy. > > Bastien, IMO, #1 applies to batch jobs as well. > > On Tue, Jan 15, 2019 at 6:27 AM bastien dine <bastien.d...@gmail.com > <mailto:bastien.d...@gmail.com>> wrote: > Hello Jamie, > > Does #1 apply to batch jobs too ? > > Regards, > > ------------------ > > Bastien DINE > Data Architect / Software Engineer / Sysadmin > bastiendine.io <http://bastiendine.io/> > > > Le lun. 14 janv. 2019 à 20:39, Jamie Grier <jgr...@lyft.com > <mailto:jgr...@lyft.com>> a écrit : > There are a lot of different ways to deploy Flink. It would be easier to > answer your question with a little more context about your use case but in > general I would advocate the following: > > 1) Don't run a "permanent" Flink cluster and then submit jobs to it. Instead > what you should do is run an "ephemeral" cluster per job if possible. This > keeps jobs completely isolated from each other which helps a lot with > understanding performance, debugging, looking at logs, etc. > 2) Given that you can do #1 and you are running on bare metal (as opposed to > in containers) then run one TM per physical machine. > > There are many ways to accomplish the above depending on your deployment > infrastructure (YARN, K8S, bare metal, VMs, etc) so it's hard to give > detailed input but general you'll have the best luck if you don't run > multiple jobs in the same TM/JVM. > > In terms of the TM memory usage you can set that up by configuring it in the > flink-conf.yaml file. The config key you are looking or is > taskmanager.heap.size: > https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#taskmanager-heap-size > > <https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#taskmanager-heap-size> > > > On Mon, Jan 14, 2019 at 8:05 AM Ethan Li <ethanopensou...@gmail.com > <mailto:ethanopensou...@gmail.com>> wrote: > Hello, > > I am setting up a standalone flink cluster and I am wondering what’s the best > way to distribute TaskManagers. Do we usually launch one TaskManager (with > many slots) per node or multiple TaskManagers per node (with smaller number > of slots per tm) ? Also with one TaskManager per node, I am seeing that TM > launches with only 30GB JVM heap by default while the node has 180 GB. Why is > it not launching with more memory since there is a lot available? > > Thank you very much! > > - Ethan
