Re: [Third-party Tool] Flink memory calculator
Hi, there, As Flink 1.11.0 released, we provide a new calculator[1] for this version. Feel free to try it and any feedback or suggestion is welcomed! [1] https://github.com/KarmaGYZ/flink-memory-calculator/blob/master/calculator-1.11.sh Best, Yangze Guo On Wed, Apr 1, 2020 at 9:45 PM Yangze Guo wrote: > > @Marta > Thanks for the tip! I'll do that. > > Best, > Yangze Guo > > On Wed, Apr 1, 2020 at 8:05 PM Marta Paes Moreira wrote: > > > > Hey, Yangze. > > > > I'd like to suggest that you submit this tool to Flink Community Pages [1]. > > That way it can get more exposure and it'll be easier for users to find it. > > > > Thanks for your contribution! > > > > [1] https://flink-packages.org/ > > > > On Tue, Mar 31, 2020 at 9:09 AM Yangze Guo wrote: > >> > >> Hi, there. > >> > >> In the latest version, the calculator supports dynamic options. You > >> could append all your dynamic options to the end of "bin/calculator.sh > >> [-h]". > >> Since "-tm" will be deprecated eventually, please replace it with > >> "-Dtaskmanager.memory.process.size=". > >> > >> Best, > >> Yangze Guo > >> > >> On Mon, Mar 30, 2020 at 12:57 PM Xintong Song > >> wrote: > >> > > >> > Hi Jeff, > >> > > >> > I think the purpose of this tool it to allow users play with the memory > >> > configurations without needing to actually deploy the Flink cluster or > >> > even have a job. For sanity checks, we currently have them in the > >> > start-up scripts (for standalone clusters) and resource managers (on > >> > K8s/Yarn/Mesos). > >> > > >> > I think it makes sense do the checks earlier, i.e. on the client side. > >> > But I'm not sure if JobListener is the right place. IIUC, JobListener is > >> > invoked before submitting a specific job, while the mentioned checks > >> > validate Flink's cluster level configurations. It might be okay for a > >> > job cluster, but does not cover the scenarios of session clusters. > >> > > >> > Thank you~ > >> > > >> > Xintong Song > >> > > >> > > >> > > >> > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: > >> >> > >> >> Thanks for your feedbacks, @Xintong and @Jeff. > >> >> > >> >> @Jeff > >> >> I think it would always be good to leverage exist logic in Flink, such > >> >> as JobListener. However, this calculator does not only target to check > >> >> the conflict, it also targets to provide the calculating result to > >> >> user before the job is actually deployed in case there is any > >> >> unexpected configuration. It's a good point that we need to parse the > >> >> dynamic configs. I prefer to parse the dynamic configs and cli > >> >> commands in bash instead of adding hook in JobListener. > >> >> > >> >> Best, > >> >> Yangze Guo > >> >> > >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > >> >> > > >> >> > Hi Yangze, > >> >> > > >> >> > Does this tool just parse the configuration in flink-conf.yaml ? > >> >> > Maybe it could be done in JobListener [1] (we should enhance it via > >> >> > adding hook before job submission), so that it could all the cases > >> >> > (e.g. parameters coming from command line) > >> >> > > >> >> > [1] > >> >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > >> >> > > >> >> > > >> >> > Yangze Guo 于2020年3月30日周一 上午9:40写道： > >> >> >> > >> >> >> Hi, Yun, > >> >> >> > >> >> >> I'm sorry that it currently could not handle it. But I think it is a > >> >> >> really good idea and that feature would be added to the next version. > >> >> >> > >> >> >> Best, > >> >> >> Yangze Guo > >> >> >> > >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > >> >> >> > > >> >> >> > Very interesting and convenient tool, just a quick question: could > >> &
Re: [Third-party Tool] Flink memory calculator
@Marta Thanks for the tip! I'll do that. Best, Yangze Guo On Wed, Apr 1, 2020 at 8:05 PM Marta Paes Moreira wrote: > > Hey, Yangze. > > I'd like to suggest that you submit this tool to Flink Community Pages [1]. > That way it can get more exposure and it'll be easier for users to find it. > > Thanks for your contribution! > > [1] https://flink-packages.org/ > > On Tue, Mar 31, 2020 at 9:09 AM Yangze Guo wrote: >> >> Hi, there. >> >> In the latest version, the calculator supports dynamic options. You >> could append all your dynamic options to the end of "bin/calculator.sh >> [-h]". >> Since "-tm" will be deprecated eventually, please replace it with >> "-Dtaskmanager.memory.process.size=". >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 12:57 PM Xintong Song wrote: >> > >> > Hi Jeff, >> > >> > I think the purpose of this tool it to allow users play with the memory >> > configurations without needing to actually deploy the Flink cluster or >> > even have a job. For sanity checks, we currently have them in the start-up >> > scripts (for standalone clusters) and resource managers (on >> > K8s/Yarn/Mesos). >> > >> > I think it makes sense do the checks earlier, i.e. on the client side. But >> > I'm not sure if JobListener is the right place. IIUC, JobListener is >> > invoked before submitting a specific job, while the mentioned checks >> > validate Flink's cluster level configurations. It might be okay for a job >> > cluster, but does not cover the scenarios of session clusters. >> > >> > Thank you~ >> > >> > Xintong Song >> > >> > >> > >> > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: >> >> >> >> Thanks for your feedbacks, @Xintong and @Jeff. >> >> >> >> @Jeff >> >> I think it would always be good to leverage exist logic in Flink, such >> >> as JobListener. However, this calculator does not only target to check >> >> the conflict, it also targets to provide the calculating result to >> >> user before the job is actually deployed in case there is any >> >> unexpected configuration. It's a good point that we need to parse the >> >> dynamic configs. I prefer to parse the dynamic configs and cli >> >> commands in bash instead of adding hook in JobListener. >> >> >> >> Best, >> >> Yangze Guo >> >> >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: >> >> > >> >> > Hi Yangze, >> >> > >> >> > Does this tool just parse the configuration in flink-conf.yaml ? Maybe >> >> > it could be done in JobListener [1] (we should enhance it via adding >> >> > hook before job submission), so that it could all the cases (e.g. >> >> > parameters coming from command line) >> >> > >> >> > [1] >> >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 >> >> > >> >> > >> >> > Yangze Guo 于2020年3月30日周一 上午9:40写道： >> >> >> >> >> >> Hi, Yun, >> >> >> >> >> >> I'm sorry that it currently could not handle it. But I think it is a >> >> >> really good idea and that feature would be added to the next version. >> >> >> >> >> >> Best, >> >> >> Yangze Guo >> >> >> >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> >> >> > >> >> >> > Very interesting and convenient tool, just a quick question: could >> >> >> > this tool also handle deployment cluster commands like "-tm" mixed >> >> >> > with configuration in `flink-conf.yaml` ? >> >> >> > >> >> >> > Best >> >> >> > Yun Tang >> >> >> > >> >> >> > From: Yangze Guo >> >> >> > Sent: Friday, March 27, 2020 18:00 >> >> >> > To: user ; user...@flink.apache.org >> >> >> > >> >> >> > Subject: [Third-party Tool] Flink memory calculator >> >> >> > >> >> >> > Hi, there. >> >> >> > >> >> >> > In r
Re: [Third-party Tool] Flink memory calculator
@Marta Thanks for the tip! I'll do that. Best, Yangze Guo On Wed, Apr 1, 2020 at 8:05 PM Marta Paes Moreira wrote: > > Hey, Yangze. > > I'd like to suggest that you submit this tool to Flink Community Pages [1]. > That way it can get more exposure and it'll be easier for users to find it. > > Thanks for your contribution! > > [1] https://flink-packages.org/ > > On Tue, Mar 31, 2020 at 9:09 AM Yangze Guo wrote: >> >> Hi, there. >> >> In the latest version, the calculator supports dynamic options. You >> could append all your dynamic options to the end of "bin/calculator.sh >> [-h]". >> Since "-tm" will be deprecated eventually, please replace it with >> "-Dtaskmanager.memory.process.size=". >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 12:57 PM Xintong Song wrote: >> > >> > Hi Jeff, >> > >> > I think the purpose of this tool it to allow users play with the memory >> > configurations without needing to actually deploy the Flink cluster or >> > even have a job. For sanity checks, we currently have them in the start-up >> > scripts (for standalone clusters) and resource managers (on >> > K8s/Yarn/Mesos). >> > >> > I think it makes sense do the checks earlier, i.e. on the client side. But >> > I'm not sure if JobListener is the right place. IIUC, JobListener is >> > invoked before submitting a specific job, while the mentioned checks >> > validate Flink's cluster level configurations. It might be okay for a job >> > cluster, but does not cover the scenarios of session clusters. >> > >> > Thank you~ >> > >> > Xintong Song >> > >> > >> > >> > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: >> >> >> >> Thanks for your feedbacks, @Xintong and @Jeff. >> >> >> >> @Jeff >> >> I think it would always be good to leverage exist logic in Flink, such >> >> as JobListener. However, this calculator does not only target to check >> >> the conflict, it also targets to provide the calculating result to >> >> user before the job is actually deployed in case there is any >> >> unexpected configuration. It's a good point that we need to parse the >> >> dynamic configs. I prefer to parse the dynamic configs and cli >> >> commands in bash instead of adding hook in JobListener. >> >> >> >> Best, >> >> Yangze Guo >> >> >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: >> >> > >> >> > Hi Yangze, >> >> > >> >> > Does this tool just parse the configuration in flink-conf.yaml ? Maybe >> >> > it could be done in JobListener [1] (we should enhance it via adding >> >> > hook before job submission), so that it could all the cases (e.g. >> >> > parameters coming from command line) >> >> > >> >> > [1] >> >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 >> >> > >> >> > >> >> > Yangze Guo 于2020年3月30日周一 上午9:40写道： >> >> >> >> >> >> Hi, Yun, >> >> >> >> >> >> I'm sorry that it currently could not handle it. But I think it is a >> >> >> really good idea and that feature would be added to the next version. >> >> >> >> >> >> Best, >> >> >> Yangze Guo >> >> >> >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> >> >> > >> >> >> > Very interesting and convenient tool, just a quick question: could >> >> >> > this tool also handle deployment cluster commands like "-tm" mixed >> >> >> > with configuration in `flink-conf.yaml` ? >> >> >> > >> >> >> > Best >> >> >> > Yun Tang >> >> >> > >> >> >> > From: Yangze Guo >> >> >> > Sent: Friday, March 27, 2020 18:00 >> >> >> > To: user ; user-zh@flink.apache.org >> >> >> > >> >> >> > Subject: [Third-party Tool] Flink memory calculator >> >> >> > >> >> >> > Hi, there. >> >> >> > >> >> >> > In r
Re: [Third-party Tool] Flink memory calculator
Hey, Yangze. I'd like to suggest that you submit this tool to Flink Community Pages [1]. That way it can get more exposure and it'll be easier for users to find it. Thanks for your contribution! [1] https://flink-packages.org/ On Tue, Mar 31, 2020 at 9:09 AM Yangze Guo wrote: > Hi, there. > > In the latest version, the calculator supports dynamic options. You > could append all your dynamic options to the end of "bin/calculator.sh > [-h]". > Since "-tm" will be deprecated eventually, please replace it with > "-Dtaskmanager.memory.process.size=". > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 12:57 PM Xintong Song > wrote: > > > > Hi Jeff, > > > > I think the purpose of this tool it to allow users play with the memory > configurations without needing to actually deploy the Flink cluster or even > have a job. For sanity checks, we currently have them in the start-up > scripts (for standalone clusters) and resource managers (on K8s/Yarn/Mesos). > > > > I think it makes sense do the checks earlier, i.e. on the client side. > But I'm not sure if JobListener is the right place. IIUC, JobListener is > invoked before submitting a specific job, while the mentioned checks > validate Flink's cluster level configurations. It might be okay for a job > cluster, but does not cover the scenarios of session clusters. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: > >> > >> Thanks for your feedbacks, @Xintong and @Jeff. > >> > >> @Jeff > >> I think it would always be good to leverage exist logic in Flink, such > >> as JobListener. However, this calculator does not only target to check > >> the conflict, it also targets to provide the calculating result to > >> user before the job is actually deployed in case there is any > >> unexpected configuration. It's a good point that we need to parse the > >> dynamic configs. I prefer to parse the dynamic configs and cli > >> commands in bash instead of adding hook in JobListener. > >> > >> Best, > >> Yangze Guo > >> > >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > >> > > >> > Hi Yangze, > >> > > >> > Does this tool just parse the configuration in flink-conf.yaml ? > Maybe it could be done in JobListener [1] (we should enhance it via adding > hook before job submission), so that it could all the cases (e.g. > parameters coming from command line) > >> > > >> > [1] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > >> > > >> > > >> > Yangze Guo 于2020年3月30日周一 上午9:40写道： > >> >> > >> >> Hi, Yun, > >> >> > >> >> I'm sorry that it currently could not handle it. But I think it is a > >> >> really good idea and that feature would be added to the next version. > >> >> > >> >> Best, > >> >> Yangze Guo > >> >> > >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > >> >> > > >> >> > Very interesting and convenient tool, just a quick question: could > this tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > >> >> > > >> >> > Best > >> >> > Yun Tang > >> >> > > >> >> > From: Yangze Guo > >> >> > Sent: Friday, March 27, 2020 18:00 > >> >> > To: user ; user...@flink.apache.org < > user...@flink.apache.org> > >> >> > Subject: [Third-party Tool] Flink memory calculator > >> >> > > >> >> > Hi, there. > >> >> > > >> >> > In release-1.10, the memory setup of task managers has changed a > lot. > >> >> > I would like to provide here a third-party tool to simulate and get > >> >> > the calculation result of Flink's memory configuration. > >> >> > > >> >> > Although there is already a detailed setup guide[1] and migration > >> >> > guide[2] officially, the calculator could further allow users to: > >> >> > - Verify if there is any conflict in their configuration. The > >> >> > calculator is more lightweight than starting a Flink cluster, > >> >> > especially when ru
Re: [Third-party Tool] Flink memory calculator
Hi, there. In the latest version, the calculator supports dynamic options. You could append all your dynamic options to the end of "bin/calculator.sh [-h]". Since "-tm" will be deprecated eventually, please replace it with "-Dtaskmanager.memory.process.size=". Best, Yangze Guo On Mon, Mar 30, 2020 at 12:57 PM Xintong Song wrote: > > Hi Jeff, > > I think the purpose of this tool it to allow users play with the memory > configurations without needing to actually deploy the Flink cluster or even > have a job. For sanity checks, we currently have them in the start-up scripts > (for standalone clusters) and resource managers (on K8s/Yarn/Mesos). > > I think it makes sense do the checks earlier, i.e. on the client side. But > I'm not sure if JobListener is the right place. IIUC, JobListener is invoked > before submitting a specific job, while the mentioned checks validate Flink's > cluster level configurations. It might be okay for a job cluster, but does > not cover the scenarios of session clusters. > > Thank you~ > > Xintong Song > > > > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: >> >> Thanks for your feedbacks, @Xintong and @Jeff. >> >> @Jeff >> I think it would always be good to leverage exist logic in Flink, such >> as JobListener. However, this calculator does not only target to check >> the conflict, it also targets to provide the calculating result to >> user before the job is actually deployed in case there is any >> unexpected configuration. It's a good point that we need to parse the >> dynamic configs. I prefer to parse the dynamic configs and cli >> commands in bash instead of adding hook in JobListener. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: >> > >> > Hi Yangze, >> > >> > Does this tool just parse the configuration in flink-conf.yaml ? Maybe it >> > could be done in JobListener [1] (we should enhance it via adding hook >> > before job submission), so that it could all the cases (e.g. parameters >> > coming from command line) >> > >> > [1] >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 >> > >> > >> > Yangze Guo 于2020年3月30日周一 上午9:40写道： >> >> >> >> Hi, Yun, >> >> >> >> I'm sorry that it currently could not handle it. But I think it is a >> >> really good idea and that feature would be added to the next version. >> >> >> >> Best, >> >> Yangze Guo >> >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> >> > >> >> > Very interesting and convenient tool, just a quick question: could this >> >> > tool also handle deployment cluster commands like "-tm" mixed with >> >> > configuration in `flink-conf.yaml` ? >> >> > >> >> > Best >> >> > Yun Tang >> >> > >> >> > From: Yangze Guo >> >> > Sent: Friday, March 27, 2020 18:00 >> >> > To: user ; user-zh@flink.apache.org >> >> > >> >> > Subject: [Third-party Tool] Flink memory calculator >> >> > >> >> > Hi, there. >> >> > >> >> > In release-1.10, the memory setup of task managers has changed a lot. >> >> > I would like to provide here a third-party tool to simulate and get >> >> > the calculation result of Flink's memory configuration. >> >> > >> >> > Although there is already a detailed setup guide[1] and migration >> >> > guide[2] officially, the calculator could further allow users to: >> >> > - Verify if there is any conflict in their configuration. The >> >> > calculator is more lightweight than starting a Flink cluster, >> >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> >> > their configuration is correct locally before deploying it to external >> >> > resource managers. >> >> > - Get all of the memory configurations before deploying. User may set >> >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> >> > But they also want to know the total memory consumption of Flink. With >> >> > this tool, users could get all of the memory configurations they are >> >> > interested in. If anything is unexpected, they would not need to >> >> > re-deploy a Flink cluster. >> >> > >> >> > The repo link of this tool is >> >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> >> > exactly the same as your Flink dist. For more details, please take a >> >> > look at the README. >> >> > >> >> > Any feedback or suggestion is welcomed! >> >> > >> >> > [1] >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html >> >> > [2] >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html >> >> > >> >> > Best, >> >> > Yangze Guo >> > >> > >> > >> > -- >> > Best Regards >> > >> > Jeff Zhang
Re: [Third-party Tool] Flink memory calculator
Hi, there. In the latest version, the calculator supports dynamic options. You could append all your dynamic options to the end of "bin/calculator.sh [-h]". Since "-tm" will be deprecated eventually, please replace it with "-Dtaskmanager.memory.process.size=". Best, Yangze Guo On Mon, Mar 30, 2020 at 12:57 PM Xintong Song wrote: > > Hi Jeff, > > I think the purpose of this tool it to allow users play with the memory > configurations without needing to actually deploy the Flink cluster or even > have a job. For sanity checks, we currently have them in the start-up scripts > (for standalone clusters) and resource managers (on K8s/Yarn/Mesos). > > I think it makes sense do the checks earlier, i.e. on the client side. But > I'm not sure if JobListener is the right place. IIUC, JobListener is invoked > before submitting a specific job, while the mentioned checks validate Flink's > cluster level configurations. It might be okay for a job cluster, but does > not cover the scenarios of session clusters. > > Thank you~ > > Xintong Song > > > > On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: >> >> Thanks for your feedbacks, @Xintong and @Jeff. >> >> @Jeff >> I think it would always be good to leverage exist logic in Flink, such >> as JobListener. However, this calculator does not only target to check >> the conflict, it also targets to provide the calculating result to >> user before the job is actually deployed in case there is any >> unexpected configuration. It's a good point that we need to parse the >> dynamic configs. I prefer to parse the dynamic configs and cli >> commands in bash instead of adding hook in JobListener. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: >> > >> > Hi Yangze, >> > >> > Does this tool just parse the configuration in flink-conf.yaml ? Maybe it >> > could be done in JobListener [1] (we should enhance it via adding hook >> > before job submission), so that it could all the cases (e.g. parameters >> > coming from command line) >> > >> > [1] >> > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 >> > >> > >> > Yangze Guo 于2020年3月30日周一 上午9:40写道： >> >> >> >> Hi, Yun, >> >> >> >> I'm sorry that it currently could not handle it. But I think it is a >> >> really good idea and that feature would be added to the next version. >> >> >> >> Best, >> >> Yangze Guo >> >> >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> >> > >> >> > Very interesting and convenient tool, just a quick question: could this >> >> > tool also handle deployment cluster commands like "-tm" mixed with >> >> > configuration in `flink-conf.yaml` ? >> >> > >> >> > Best >> >> > Yun Tang >> >> > >> >> > From: Yangze Guo >> >> > Sent: Friday, March 27, 2020 18:00 >> >> > To: user ; user...@flink.apache.org >> >> > >> >> > Subject: [Third-party Tool] Flink memory calculator >> >> > >> >> > Hi, there. >> >> > >> >> > In release-1.10, the memory setup of task managers has changed a lot. >> >> > I would like to provide here a third-party tool to simulate and get >> >> > the calculation result of Flink's memory configuration. >> >> > >> >> > Although there is already a detailed setup guide[1] and migration >> >> > guide[2] officially, the calculator could further allow users to: >> >> > - Verify if there is any conflict in their configuration. The >> >> > calculator is more lightweight than starting a Flink cluster, >> >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> >> > their configuration is correct locally before deploying it to external >> >> > resource managers. >> >> > - Get all of the memory configurations before deploying. User may set >> >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> >> > But they also want to know the total memory consumption of Flink. With >> >> > this tool, users could get all of the memory configurations they are >> >> > interested in. If anything is unexpected, they would not need to >> >> > re-deploy a Flink cluster. >> >> > >> >> > The repo link of this tool is >> >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> >> > exactly the same as your Flink dist. For more details, please take a >> >> > look at the README. >> >> > >> >> > Any feedback or suggestion is welcomed! >> >> > >> >> > [1] >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html >> >> > [2] >> >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html >> >> > >> >> > Best, >> >> > Yangze Guo >> > >> > >> > >> > -- >> > Best Regards >> > >> > Jeff Zhang
Re: [Third-party Tool] Flink memory calculator
Hi Jeff, I think the purpose of this tool it to allow users play with the memory configurations without needing to actually deploy the Flink cluster or even have a job. For sanity checks, we currently have them in the start-up scripts (for standalone clusters) and resource managers (on K8s/Yarn/Mesos). I think it makes sense do the checks earlier, i.e. on the client side. But I'm not sure if JobListener is the right place. IIUC, JobListener is invoked before submitting a specific job, while the mentioned checks validate Flink's cluster level configurations. It might be okay for a job cluster, but does not cover the scenarios of session clusters. Thank you~ Xintong Song On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: > Thanks for your feedbacks, @Xintong and @Jeff. > > @Jeff > I think it would always be good to leverage exist logic in Flink, such > as JobListener. However, this calculator does not only target to check > the conflict, it also targets to provide the calculating result to > user before the job is actually deployed in case there is any > unexpected configuration. It's a good point that we need to parse the > dynamic configs. I prefer to parse the dynamic configs and cli > commands in bash instead of adding hook in JobListener. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > > > > Hi Yangze, > > > > Does this tool just parse the configuration in flink-conf.yaml ? Maybe > it could be done in JobListener [1] (we should enhance it via adding hook > before job submission), so that it could all the cases (e.g. parameters > coming from command line) > > > > [1] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > > > > > > Yangze Guo 于2020年3月30日周一 上午9:40写道： > >> > >> Hi, Yun, > >> > >> I'm sorry that it currently could not handle it. But I think it is a > >> really good idea and that feature would be added to the next version. > >> > >> Best, > >> Yangze Guo > >> > >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > >> > > >> > Very interesting and convenient tool, just a quick question: could > this tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > >> > > >> > Best > >> > Yun Tang > >> > > >> > From: Yangze Guo > >> > Sent: Friday, March 27, 2020 18:00 > >> > To: user ; user...@flink.apache.org < > user...@flink.apache.org> > >> > Subject: [Third-party Tool] Flink memory calculator > >> > > >> > Hi, there. > >> > > >> > In release-1.10, the memory setup of task managers has changed a lot. > >> > I would like to provide here a third-party tool to simulate and get > >> > the calculation result of Flink's memory configuration. > >> > > >> > Although there is already a detailed setup guide[1] and migration > >> > guide[2] officially, the calculator could further allow users to: > >> > - Verify if there is any conflict in their configuration. The > >> > calculator is more lightweight than starting a Flink cluster, > >> > especially when running Flink on Yarn/Kubernetes. User could make sure > >> > their configuration is correct locally before deploying it to external > >> > resource managers. > >> > - Get all of the memory configurations before deploying. User may set > >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > >> > But they also want to know the total memory consumption of Flink. With > >> > this tool, users could get all of the memory configurations they are > >> > interested in. If anything is unexpected, they would not need to > >> > re-deploy a Flink cluster. > >> > > >> > The repo link of this tool is > >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > >> > BashJavaUtils.jar of Flink and ensures the calculation result is > >> > exactly the same as your Flink dist. For more details, please take a > >> > look at the README. > >> > > >> > Any feedback or suggestion is welcomed! > >> > > >> > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > >> > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > >> > > >> > Best, > >> > Yangze Guo > > > > > > > > -- > > Best Regards > > > > Jeff Zhang >
Re: [Third-party Tool] Flink memory calculator
Hi Jeff, I think the purpose of this tool it to allow users play with the memory configurations without needing to actually deploy the Flink cluster or even have a job. For sanity checks, we currently have them in the start-up scripts (for standalone clusters) and resource managers (on K8s/Yarn/Mesos). I think it makes sense do the checks earlier, i.e. on the client side. But I'm not sure if JobListener is the right place. IIUC, JobListener is invoked before submitting a specific job, while the mentioned checks validate Flink's cluster level configurations. It might be okay for a job cluster, but does not cover the scenarios of session clusters. Thank you~ Xintong Song On Mon, Mar 30, 2020 at 12:03 PM Yangze Guo wrote: > Thanks for your feedbacks, @Xintong and @Jeff. > > @Jeff > I think it would always be good to leverage exist logic in Flink, such > as JobListener. However, this calculator does not only target to check > the conflict, it also targets to provide the calculating result to > user before the job is actually deployed in case there is any > unexpected configuration. It's a good point that we need to parse the > dynamic configs. I prefer to parse the dynamic configs and cli > commands in bash instead of adding hook in JobListener. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > > > > Hi Yangze, > > > > Does this tool just parse the configuration in flink-conf.yaml ? Maybe > it could be done in JobListener [1] (we should enhance it via adding hook > before job submission), so that it could all the cases (e.g. parameters > coming from command line) > > > > [1] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > > > > > > Yangze Guo 于2020年3月30日周一 上午9:40写道： > >> > >> Hi, Yun, > >> > >> I'm sorry that it currently could not handle it. But I think it is a > >> really good idea and that feature would be added to the next version. > >> > >> Best, > >> Yangze Guo > >> > >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > >> > > >> > Very interesting and convenient tool, just a quick question: could > this tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > >> > > >> > Best > >> > Yun Tang > >> > > >> > From: Yangze Guo > >> > Sent: Friday, March 27, 2020 18:00 > >> > To: user ; user-zh@flink.apache.org < > user-zh@flink.apache.org> > >> > Subject: [Third-party Tool] Flink memory calculator > >> > > >> > Hi, there. > >> > > >> > In release-1.10, the memory setup of task managers has changed a lot. > >> > I would like to provide here a third-party tool to simulate and get > >> > the calculation result of Flink's memory configuration. > >> > > >> > Although there is already a detailed setup guide[1] and migration > >> > guide[2] officially, the calculator could further allow users to: > >> > - Verify if there is any conflict in their configuration. The > >> > calculator is more lightweight than starting a Flink cluster, > >> > especially when running Flink on Yarn/Kubernetes. User could make sure > >> > their configuration is correct locally before deploying it to external > >> > resource managers. > >> > - Get all of the memory configurations before deploying. User may set > >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > >> > But they also want to know the total memory consumption of Flink. With > >> > this tool, users could get all of the memory configurations they are > >> > interested in. If anything is unexpected, they would not need to > >> > re-deploy a Flink cluster. > >> > > >> > The repo link of this tool is > >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > >> > BashJavaUtils.jar of Flink and ensures the calculation result is > >> > exactly the same as your Flink dist. For more details, please take a > >> > look at the README. > >> > > >> > Any feedback or suggestion is welcomed! > >> > > >> > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > >> > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > >> > > >> > Best, > >> > Yangze Guo > > > > > > > > -- > > Best Regards > > > > Jeff Zhang >
Re: [Third-party Tool] Flink memory calculator
Thanks for your feedbacks, @Xintong and @Jeff. @Jeff I think it would always be good to leverage exist logic in Flink, such as JobListener. However, this calculator does not only target to check the conflict, it also targets to provide the calculating result to user before the job is actually deployed in case there is any unexpected configuration. It's a good point that we need to parse the dynamic configs. I prefer to parse the dynamic configs and cli commands in bash instead of adding hook in JobListener. Best, Yangze Guo On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > > Hi Yangze, > > Does this tool just parse the configuration in flink-conf.yaml ? Maybe it > could be done in JobListener [1] (we should enhance it via adding hook before > job submission), so that it could all the cases (e.g. parameters coming from > command line) > > [1] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > > > Yangze Guo 于2020年3月30日周一 上午9:40写道： >> >> Hi, Yun, >> >> I'm sorry that it currently could not handle it. But I think it is a >> really good idea and that feature would be added to the next version. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> > >> > Very interesting and convenient tool, just a quick question: could this >> > tool also handle deployment cluster commands like "-tm" mixed with >> > configuration in `flink-conf.yaml` ? >> > >> > Best >> > Yun Tang >> > ________ >> > From: Yangze Guo >> > Sent: Friday, March 27, 2020 18:00 >> > To: user ; user-zh@flink.apache.org >> > >> > Subject: [Third-party Tool] Flink memory calculator >> > >> > Hi, there. >> > >> > In release-1.10, the memory setup of task managers has changed a lot. >> > I would like to provide here a third-party tool to simulate and get >> > the calculation result of Flink's memory configuration. >> > >> > Although there is already a detailed setup guide[1] and migration >> > guide[2] officially, the calculator could further allow users to: >> > - Verify if there is any conflict in their configuration. The >> > calculator is more lightweight than starting a Flink cluster, >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> > their configuration is correct locally before deploying it to external >> > resource managers. >> > - Get all of the memory configurations before deploying. User may set >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> > But they also want to know the total memory consumption of Flink. With >> > this tool, users could get all of the memory configurations they are >> > interested in. If anything is unexpected, they would not need to >> > re-deploy a Flink cluster. >> > >> > The repo link of this tool is >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> > exactly the same as your Flink dist. For more details, please take a >> > look at the README. >> > >> > Any feedback or suggestion is welcomed! >> > >> > [1] >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html >> > [2] >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html >> > >> > Best, >> > Yangze Guo > > > > -- > Best Regards > > Jeff Zhang
Re: [Third-party Tool] Flink memory calculator
Thanks for your feedbacks, @Xintong and @Jeff. @Jeff I think it would always be good to leverage exist logic in Flink, such as JobListener. However, this calculator does not only target to check the conflict, it also targets to provide the calculating result to user before the job is actually deployed in case there is any unexpected configuration. It's a good point that we need to parse the dynamic configs. I prefer to parse the dynamic configs and cli commands in bash instead of adding hook in JobListener. Best, Yangze Guo On Mon, Mar 30, 2020 at 10:32 AM Jeff Zhang wrote: > > Hi Yangze, > > Does this tool just parse the configuration in flink-conf.yaml ? Maybe it > could be done in JobListener [1] (we should enhance it via adding hook before > job submission), so that it could all the cases (e.g. parameters coming from > command line) > > [1] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 > > > Yangze Guo 于2020年3月30日周一 上午9:40写道： >> >> Hi, Yun, >> >> I'm sorry that it currently could not handle it. But I think it is a >> really good idea and that feature would be added to the next version. >> >> Best, >> Yangze Guo >> >> On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: >> > >> > Very interesting and convenient tool, just a quick question: could this >> > tool also handle deployment cluster commands like "-tm" mixed with >> > configuration in `flink-conf.yaml` ? >> > >> > Best >> > Yun Tang >> > ________ >> > From: Yangze Guo >> > Sent: Friday, March 27, 2020 18:00 >> > To: user ; user...@flink.apache.org >> > >> > Subject: [Third-party Tool] Flink memory calculator >> > >> > Hi, there. >> > >> > In release-1.10, the memory setup of task managers has changed a lot. >> > I would like to provide here a third-party tool to simulate and get >> > the calculation result of Flink's memory configuration. >> > >> > Although there is already a detailed setup guide[1] and migration >> > guide[2] officially, the calculator could further allow users to: >> > - Verify if there is any conflict in their configuration. The >> > calculator is more lightweight than starting a Flink cluster, >> > especially when running Flink on Yarn/Kubernetes. User could make sure >> > their configuration is correct locally before deploying it to external >> > resource managers. >> > - Get all of the memory configurations before deploying. User may set >> > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. >> > But they also want to know the total memory consumption of Flink. With >> > this tool, users could get all of the memory configurations they are >> > interested in. If anything is unexpected, they would not need to >> > re-deploy a Flink cluster. >> > >> > The repo link of this tool is >> > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the >> > BashJavaUtils.jar of Flink and ensures the calculation result is >> > exactly the same as your Flink dist. For more details, please take a >> > look at the README. >> > >> > Any feedback or suggestion is welcomed! >> > >> > [1] >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html >> > [2] >> > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html >> > >> > Best, >> > Yangze Guo > > > > -- > Best Regards > > Jeff Zhang
Re: [Third-party Tool] Flink memory calculator
Hi Yangze, Does this tool just parse the configuration in flink-conf.yaml ? Maybe it could be done in JobListener [1] (we should enhance it via adding hook before job submission), so that it could all the cases (e.g. parameters coming from command line) [1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 Yangze Guo 于2020年3月30日周一 上午9:40写道： > Hi, Yun, > > I'm sorry that it currently could not handle it. But I think it is a > really good idea and that feature would be added to the next version. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > > > Very interesting and convenient tool, just a quick question: could this > tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > > > > Best > > Yun Tang > > > > From: Yangze Guo > > Sent: Friday, March 27, 2020 18:00 > > To: user ; user...@flink.apache.org < > user...@flink.apache.org> > > Subject: [Third-party Tool] Flink memory calculator > > > > Hi, there. > > > > In release-1.10, the memory setup of task managers has changed a lot. > > I would like to provide here a third-party tool to simulate and get > > the calculation result of Flink's memory configuration. > > > > Although there is already a detailed setup guide[1] and migration > > guide[2] officially, the calculator could further allow users to: > > - Verify if there is any conflict in their configuration. The > > calculator is more lightweight than starting a Flink cluster, > > especially when running Flink on Yarn/Kubernetes. User could make sure > > their configuration is correct locally before deploying it to external > > resource managers. > > - Get all of the memory configurations before deploying. User may set > > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > > But they also want to know the total memory consumption of Flink. With > > this tool, users could get all of the memory configurations they are > > interested in. If anything is unexpected, they would not need to > > re-deploy a Flink cluster. > > > > The repo link of this tool is > > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > > BashJavaUtils.jar of Flink and ensures the calculation result is > > exactly the same as your Flink dist. For more details, please take a > > look at the README. > > > > Any feedback or suggestion is welcomed! > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > > > Best, > > Yangze Guo > -- Best Regards Jeff Zhang
Re: [Third-party Tool] Flink memory calculator
Hi Yangze, Does this tool just parse the configuration in flink-conf.yaml ? Maybe it could be done in JobListener [1] (we should enhance it via adding hook before job submission), so that it could all the cases (e.g. parameters coming from command line) [1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java#L35 Yangze Guo 于2020年3月30日周一 上午9:40写道： > Hi, Yun, > > I'm sorry that it currently could not handle it. But I think it is a > really good idea and that feature would be added to the next version. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > > > Very interesting and convenient tool, just a quick question: could this > tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > > > > Best > > Yun Tang > > > > From: Yangze Guo > > Sent: Friday, March 27, 2020 18:00 > > To: user ; user-zh@flink.apache.org < > user-zh@flink.apache.org> > > Subject: [Third-party Tool] Flink memory calculator > > > > Hi, there. > > > > In release-1.10, the memory setup of task managers has changed a lot. > > I would like to provide here a third-party tool to simulate and get > > the calculation result of Flink's memory configuration. > > > > Although there is already a detailed setup guide[1] and migration > > guide[2] officially, the calculator could further allow users to: > > - Verify if there is any conflict in their configuration. The > > calculator is more lightweight than starting a Flink cluster, > > especially when running Flink on Yarn/Kubernetes. User could make sure > > their configuration is correct locally before deploying it to external > > resource managers. > > - Get all of the memory configurations before deploying. User may set > > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > > But they also want to know the total memory consumption of Flink. With > > this tool, users could get all of the memory configurations they are > > interested in. If anything is unexpected, they would not need to > > re-deploy a Flink cluster. > > > > The repo link of this tool is > > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > > BashJavaUtils.jar of Flink and ensures the calculation result is > > exactly the same as your Flink dist. For more details, please take a > > look at the README. > > > > Any feedback or suggestion is welcomed! > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > > > Best, > > Yangze Guo > -- Best Regards Jeff Zhang
Re: [Third-party Tool] Flink memory calculator
Thanks Yangze, I've tried the tool and I think its very helpful. Thank you~ Xintong Song On Mon, Mar 30, 2020 at 9:40 AM Yangze Guo wrote: > Hi, Yun, > > I'm sorry that it currently could not handle it. But I think it is a > really good idea and that feature would be added to the next version. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > > > Very interesting and convenient tool, just a quick question: could this > tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > > > > Best > > Yun Tang > > > > From: Yangze Guo > > Sent: Friday, March 27, 2020 18:00 > > To: user ; user...@flink.apache.org < > user...@flink.apache.org> > > Subject: [Third-party Tool] Flink memory calculator > > > > Hi, there. > > > > In release-1.10, the memory setup of task managers has changed a lot. > > I would like to provide here a third-party tool to simulate and get > > the calculation result of Flink's memory configuration. > > > > Although there is already a detailed setup guide[1] and migration > > guide[2] officially, the calculator could further allow users to: > > - Verify if there is any conflict in their configuration. The > > calculator is more lightweight than starting a Flink cluster, > > especially when running Flink on Yarn/Kubernetes. User could make sure > > their configuration is correct locally before deploying it to external > > resource managers. > > - Get all of the memory configurations before deploying. User may set > > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > > But they also want to know the total memory consumption of Flink. With > > this tool, users could get all of the memory configurations they are > > interested in. If anything is unexpected, they would not need to > > re-deploy a Flink cluster. > > > > The repo link of this tool is > > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > > BashJavaUtils.jar of Flink and ensures the calculation result is > > exactly the same as your Flink dist. For more details, please take a > > look at the README. > > > > Any feedback or suggestion is welcomed! > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > > > Best, > > Yangze Guo >
Re: [Third-party Tool] Flink memory calculator
Thanks Yangze, I've tried the tool and I think its very helpful. Thank you~ Xintong Song On Mon, Mar 30, 2020 at 9:40 AM Yangze Guo wrote: > Hi, Yun, > > I'm sorry that it currently could not handle it. But I think it is a > really good idea and that feature would be added to the next version. > > Best, > Yangze Guo > > On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > > > Very interesting and convenient tool, just a quick question: could this > tool also handle deployment cluster commands like "-tm" mixed with > configuration in `flink-conf.yaml` ? > > > > Best > > Yun Tang > > > > From: Yangze Guo > > Sent: Friday, March 27, 2020 18:00 > > To: user ; user-zh@flink.apache.org < > user-zh@flink.apache.org> > > Subject: [Third-party Tool] Flink memory calculator > > > > Hi, there. > > > > In release-1.10, the memory setup of task managers has changed a lot. > > I would like to provide here a third-party tool to simulate and get > > the calculation result of Flink's memory configuration. > > > > Although there is already a detailed setup guide[1] and migration > > guide[2] officially, the calculator could further allow users to: > > - Verify if there is any conflict in their configuration. The > > calculator is more lightweight than starting a Flink cluster, > > especially when running Flink on Yarn/Kubernetes. User could make sure > > their configuration is correct locally before deploying it to external > > resource managers. > > - Get all of the memory configurations before deploying. User may set > > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > > But they also want to know the total memory consumption of Flink. With > > this tool, users could get all of the memory configurations they are > > interested in. If anything is unexpected, they would not need to > > re-deploy a Flink cluster. > > > > The repo link of this tool is > > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > > BashJavaUtils.jar of Flink and ensures the calculation result is > > exactly the same as your Flink dist. For more details, please take a > > look at the README. > > > > Any feedback or suggestion is welcomed! > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > > > Best, > > Yangze Guo >
Re: [Third-party Tool] Flink memory calculator
Hi, Yun, I'm sorry that it currently could not handle it. But I think it is a really good idea and that feature would be added to the next version. Best, Yangze Guo On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > Very interesting and convenient tool, just a quick question: could this tool > also handle deployment cluster commands like "-tm" mixed with configuration > in `flink-conf.yaml` ? > > Best > Yun Tang > > From: Yangze Guo > Sent: Friday, March 27, 2020 18:00 > To: user ; user...@flink.apache.org > > Subject: [Third-party Tool] Flink memory calculator > > Hi, there. > > In release-1.10, the memory setup of task managers has changed a lot. > I would like to provide here a third-party tool to simulate and get > the calculation result of Flink's memory configuration. > > Although there is already a detailed setup guide[1] and migration > guide[2] officially, the calculator could further allow users to: > - Verify if there is any conflict in their configuration. The > calculator is more lightweight than starting a Flink cluster, > especially when running Flink on Yarn/Kubernetes. User could make sure > their configuration is correct locally before deploying it to external > resource managers. > - Get all of the memory configurations before deploying. User may set > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > But they also want to know the total memory consumption of Flink. With > this tool, users could get all of the memory configurations they are > interested in. If anything is unexpected, they would not need to > re-deploy a Flink cluster. > > The repo link of this tool is > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > BashJavaUtils.jar of Flink and ensures the calculation result is > exactly the same as your Flink dist. For more details, please take a > look at the README. > > Any feedback or suggestion is welcomed! > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > Best, > Yangze Guo
Re: [Third-party Tool] Flink memory calculator
Hi, Yun, I'm sorry that it currently could not handle it. But I think it is a really good idea and that feature would be added to the next version. Best, Yangze Guo On Mon, Mar 30, 2020 at 12:21 AM Yun Tang wrote: > > Very interesting and convenient tool, just a quick question: could this tool > also handle deployment cluster commands like "-tm" mixed with configuration > in `flink-conf.yaml` ? > > Best > Yun Tang > > From: Yangze Guo > Sent: Friday, March 27, 2020 18:00 > To: user ; user-zh@flink.apache.org > > Subject: [Third-party Tool] Flink memory calculator > > Hi, there. > > In release-1.10, the memory setup of task managers has changed a lot. > I would like to provide here a third-party tool to simulate and get > the calculation result of Flink's memory configuration. > > Although there is already a detailed setup guide[1] and migration > guide[2] officially, the calculator could further allow users to: > - Verify if there is any conflict in their configuration. The > calculator is more lightweight than starting a Flink cluster, > especially when running Flink on Yarn/Kubernetes. User could make sure > their configuration is correct locally before deploying it to external > resource managers. > - Get all of the memory configurations before deploying. User may set > taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. > But they also want to know the total memory consumption of Flink. With > this tool, users could get all of the memory configurations they are > interested in. If anything is unexpected, they would not need to > re-deploy a Flink cluster. > > The repo link of this tool is > https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the > BashJavaUtils.jar of Flink and ensures the calculation result is > exactly the same as your Flink dist. For more details, please take a > look at the README. > > Any feedback or suggestion is welcomed! > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html > [2] > https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html > > Best, > Yangze Guo
Re: [Third-party Tool] Flink memory calculator
Very interesting and convenient tool, just a quick question: could this tool also handle deployment cluster commands like "-tm" mixed with configuration in `flink-conf.yaml` ? Best Yun Tang From: Yangze Guo Sent: Friday, March 27, 2020 18:00 To: user ; user-zh@flink.apache.org Subject: [Third-party Tool] Flink memory calculator Hi, there. In release-1.10, the memory setup of task managers has changed a lot. I would like to provide here a third-party tool to simulate and get the calculation result of Flink's memory configuration. Although there is already a detailed setup guide[1] and migration guide[2] officially, the calculator could further allow users to: - Verify if there is any conflict in their configuration. The calculator is more lightweight than starting a Flink cluster, especially when running Flink on Yarn/Kubernetes. User could make sure their configuration is correct locally before deploying it to external resource managers. - Get all of the memory configurations before deploying. User may set taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. But they also want to know the total memory consumption of Flink. With this tool, users could get all of the memory configurations they are interested in. If anything is unexpected, they would not need to re-deploy a Flink cluster. The repo link of this tool is https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the BashJavaUtils.jar of Flink and ensures the calculation result is exactly the same as your Flink dist. For more details, please take a look at the README. Any feedback or suggestion is welcomed! [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html Best, Yangze Guo
Re: [Third-party Tool] Flink memory calculator
Very interesting and convenient tool, just a quick question: could this tool also handle deployment cluster commands like "-tm" mixed with configuration in `flink-conf.yaml` ? Best Yun Tang From: Yangze Guo Sent: Friday, March 27, 2020 18:00 To: user ; user...@flink.apache.org Subject: [Third-party Tool] Flink memory calculator Hi, there. In release-1.10, the memory setup of task managers has changed a lot. I would like to provide here a third-party tool to simulate and get the calculation result of Flink's memory configuration. Although there is already a detailed setup guide[1] and migration guide[2] officially, the calculator could further allow users to: - Verify if there is any conflict in their configuration. The calculator is more lightweight than starting a Flink cluster, especially when running Flink on Yarn/Kubernetes. User could make sure their configuration is correct locally before deploying it to external resource managers. - Get all of the memory configurations before deploying. User may set taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. But they also want to know the total memory consumption of Flink. With this tool, users could get all of the memory configurations they are interested in. If anything is unexpected, they would not need to re-deploy a Flink cluster. The repo link of this tool is https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the BashJavaUtils.jar of Flink and ensures the calculation result is exactly the same as your Flink dist. For more details, please take a look at the README. Any feedback or suggestion is welcomed! [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html Best, Yangze Guo
[Third-party Tool] Flink memory calculator
Hi, there. In release-1.10, the memory setup of task managers has changed a lot. I would like to provide here a third-party tool to simulate and get the calculation result of Flink's memory configuration. Although there is already a detailed setup guide[1] and migration guide[2] officially, the calculator could further allow users to: - Verify if there is any conflict in their configuration. The calculator is more lightweight than starting a Flink cluster, especially when running Flink on Yarn/Kubernetes. User could make sure their configuration is correct locally before deploying it to external resource managers. - Get all of the memory configurations before deploying. User may set taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. But they also want to know the total memory consumption of Flink. With this tool, users could get all of the memory configurations they are interested in. If anything is unexpected, they would not need to re-deploy a Flink cluster. The repo link of this tool is https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the BashJavaUtils.jar of Flink and ensures the calculation result is exactly the same as your Flink dist. For more details, please take a look at the README. Any feedback or suggestion is welcomed! [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html Best, Yangze Guo
[Third-party Tool] Flink memory calculator
Hi, there. In release-1.10, the memory setup of task managers has changed a lot. I would like to provide here a third-party tool to simulate and get the calculation result of Flink's memory configuration. Although there is already a detailed setup guide[1] and migration guide[2] officially, the calculator could further allow users to: - Verify if there is any conflict in their configuration. The calculator is more lightweight than starting a Flink cluster, especially when running Flink on Yarn/Kubernetes. User could make sure their configuration is correct locally before deploying it to external resource managers. - Get all of the memory configurations before deploying. User may set taskmanager.memory.task.heap.size and taskmanager.memory.managed.size. But they also want to know the total memory consumption of Flink. With this tool, users could get all of the memory configurations they are interested in. If anything is unexpected, they would not need to re-deploy a Flink cluster. The repo link of this tool is https://github.com/KarmaGYZ/flink-memory-calculator. It reuses the BashJavaUtils.jar of Flink and ensures the calculation result is exactly the same as your Flink dist. For more details, please take a look at the README. Any feedback or suggestion is welcomed! [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html Best, Yangze Guo
