Thanks for your feedbacks, @Xintong and @Jeff. @Jeff I think it would always be good to leverage exist logic in Flink, such as JobListener. However, this calculator does not only target to check the conflict, it also targets to provide the calculating result to user before the job is actually deployed in case there is any unexpected configuration. It's a good point that we need to parse the dynamic configs. I prefer to parse the dynamic configs and cli commands in bash instead of adding hook in JobListener.
Best, Yangze Guo On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang <zjf...@gmail.com> wrote: > > Hi Yangze, > > Does this tool just parse the configuration in flink-conf.yaml ? Maybe it > could be done in JobListener [1] (we should enhance it via adding hook before > job submission), so that it could all the cases (e.g. parameters coming from > command line) > > [1] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > > > Yangze Guo <karma...@gmail.com> 于2020年3月30日周一 上午9:40写道： >> >> Hi, Yun, >> >> I'm sorry that it currently could not handle it. But I think it is a >> really good idea and that feature would be added to the next version. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang <myas...@live.com> wrote: >> > >> > Very interesting and convenient tool, just a quick question: could this >> > tool also handle deployment cluster commands like "-tm" mixed with >> > configuration in `flink-conf.yaml` ? >> > >> > Best >> > Yun Tang >> > ________________________________ >> > From: Yangze Guo <karma...@gmail.com> >> > Sent: Friday, March 27, 2020 18:00 >> > To: user <u...@flink.apache.org>; user-zh@flink.apache.org >> > <user-zh@flink.apache.org> >> > Subject: [Third-party Tool] Flink memory calculator >> > >> > Hi, there. >> > >> > In release-1.10, the memory setup of task managers has changed a lot. >> > I would like to provide here a third-party tool to simulate and get >> > the calculation result of Flink's memory configuration. >> > >> > Although there is already a detailed setup guide[1] and migration >> > guide[2] officially, the calculator could further allow users to: >> > - Verify if there is any conflict in their configuration. The >> > calculator is more lightweight than starting a Flink cluster, >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> > their configuration is correct locally before deploying it to external >> > resource managers. >> > - Get all of the memory configurations before deploying. User may set >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> > But they also want to know the total memory consumption of Flink. With >> > this tool, users could get all of the memory configurations they are >> > interested in. If anything is unexpected, they would not need to >> > re-deploy a Flink cluster. >> > >> > The repo link of this tool is >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> > exactly the same as your Flink dist. For more details, please take a >> > look at the README. >> > >> > Any feedback or suggestion is welcomed! >> > >> > [1] >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html >> > [2] >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html >> > >> > Best, >> > Yangze Guo > > > > -- > Best Regards > > Jeff Zhang
