Hi, there. In the latest version, the calculator supports dynamic options. You could append all your dynamic options to the end of "bin/calculator.sh [-h]". Since "-tm" will be deprecated eventually, please replace it with "-Dtaskmanager.memory.process.size=".
Best, Yangze Guo On Mon, Mar 30, 2020 at 12:57 PM Xintong Song <tonysong...@gmail.com> wrote: > > Hi Jeff, > > I think the purpose of this tool it to allow users play with the memory > configurations without needing to actually deploy the Flink cluster or even > have a job. For sanity checks, we currently have them in the start-up scripts > (for standalone clusters) and resource managers (on K8s/Yarn/Mesos). > > I think it makes sense do the checks earlier, i.e. on the client side. But > I'm not sure if JobListener is the right place. IIUC, JobListener is invoked > before submitting a specific job, while the mentioned checks validate Flink's > cluster level configurations. It might be okay for a job cluster, but does > not cover the scenarios of session clusters. > > Thank you~ > > Xintong Song > > > > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo <karma...@gmail.com> wrote: >> >> Thanks for your feedbacks, @Xintong and @Jeff. >> >> @Jeff >> I think it would always be good to leverage exist logic in Flink, such >> as JobListener. However, this calculator does not only target to check >> the conflict, it also targets to provide the calculating result to >> user before the job is actually deployed in case there is any >> unexpected configuration. It's a good point that we need to parse the >> dynamic configs. I prefer to parse the dynamic configs and cli >> commands in bash instead of adding hook in JobListener. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang <zjf...@gmail.com> wrote: >> > >> > Hi Yangze, >> > >> > Does this tool just parse the configuration in flink-conf.yaml ? Maybe it >> > could be done in JobListener [1] (we should enhance it via adding hook >> > before job submission), so that it could all the cases (e.g. parameters >> > coming from command line) >> > >> > [1] >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 >> > >> > >> > Yangze Guo <karma...@gmail.com> 于2020年3月30日周一 上午9:40写道： >> >> >> >> Hi, Yun, >> >> >> >> I'm sorry that it currently could not handle it. But I think it is a >> >> really good idea and that feature would be added to the next version. >> >> >> >> Best, >> >> Yangze Guo >> >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang <myas...@live.com> wrote: >> >> > >> >> > Very interesting and convenient tool, just a quick question: could this >> >> > tool also handle deployment cluster commands like "-tm" mixed with >> >> > configuration in `flink-conf.yaml` ? >> >> > >> >> > Best >> >> > Yun Tang >> >> > ________________________________ >> >> > From: Yangze Guo <karma...@gmail.com> >> >> > Sent: Friday, March 27, 2020 18:00 >> >> > To: user <user@flink.apache.org>; user...@flink.apache.org >> >> > <user...@flink.apache.org> >> >> > Subject: [Third-party Tool] Flink memory calculator >> >> > >> >> > Hi, there. >> >> > >> >> > In release-1.10, the memory setup of task managers has changed a lot. >> >> > I would like to provide here a third-party tool to simulate and get >> >> > the calculation result of Flink's memory configuration. >> >> > >> >> > Although there is already a detailed setup guide[1] and migration >> >> > guide[2] officially, the calculator could further allow users to: >> >> > - Verify if there is any conflict in their configuration. The >> >> > calculator is more lightweight than starting a Flink cluster, >> >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> >> > their configuration is correct locally before deploying it to external >> >> > resource managers. >> >> > - Get all of the memory configurations before deploying. User may set >> >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> >> > But they also want to know the total memory consumption of Flink. With >> >> > this tool, users could get all of the memory configurations they are >> >> > interested in. If anything is unexpected, they would not need to >> >> > re-deploy a Flink cluster. >> >> > >> >> > The repo link of this tool is >> >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> >> > exactly the same as your Flink dist. For more details, please take a >> >> > look at the README. >> >> > >> >> > Any feedback or suggestion is welcomed! >> >> > >> >> > [1] >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html >> >> > [2] >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html >> >> > >> >> > Best, >> >> > Yangze Guo >> > >> > >> > >> > -- >> > Best Regards >> > >> > Jeff Zhang
