Re: Flink configuration at runtime
Hi Amran, thanks a lot for your message. I think this is a valid feature request. I've created a JIRA ticket to track it: https://issues.apache.org/jira/browse/FLINK-14856 (this does not mean this gets addressed immediately. However, there are currently quite some improvements to the configuration system in Flink, as part of FLIP-59 and FLIP-81) Best, Robert On Tue, Nov 19, 2019 at 4:09 AM vino yang wrote: > Hi Amran, > > Change the config option at runtime? No, Flink does not support this > feature currently. > > However, for Flink on Yarn job cluster mode, you can specify different > config options for different jobs via program or flink-conf.yaml(copy a new > flink binary package then change config file). > > Best, > Vino > > amran dean 于2019年11月19日周二 上午5:53写道: > >> Is it possible to configure certain settings at runtime, on a per-job >> basis rather than globally within flink-conf.yaml? >> >> For example, I have a job where it's desirable to retain a large number >> of checkpoints via >> state.checkpoints.num-retained. >> >> The checkpoints are cheap, and it's low cost. For other jobs, I don't >> want such a large number. >> >> >>
Re: Flink configuration at runtime
Hi Amran, Change the config option at runtime? No, Flink does not support this feature currently. However, for Flink on Yarn job cluster mode, you can specify different config options for different jobs via program or flink-conf.yaml(copy a new flink binary package then change config file). Best, Vino amran dean 于2019年11月19日周二 上午5:53写道: > Is it possible to configure certain settings at runtime, on a per-job > basis rather than globally within flink-conf.yaml? > > For example, I have a job where it's desirable to retain a large number of > checkpoints via > state.checkpoints.num-retained. > > The checkpoints are cheap, and it's low cost. For other jobs, I don't want > such a large number. > > >
Re: Flink configuration at runtime
Hi Amran, Some configs, including "state.checkpoints.num-retained", are cluster configs that always apply to the entire Flink cluster. An alternative is to use per-job mode if you are running Flink jobs on k8s/docker or yarn. Thus to create a Flink cluster for a single job. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/docker.html#flink-job-cluster [2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn Thanks, Zhu Zhu amran dean 于2019年11月19日周二 上午5:53写道: > Is it possible to configure certain settings at runtime, on a per-job > basis rather than globally within flink-conf.yaml? > > For example, I have a job where it's desirable to retain a large number of > checkpoints via > state.checkpoints.num-retained. > > The checkpoints are cheap, and it's low cost. For other jobs, I don't want > such a large number. > > >
Flink configuration at runtime
Is it possible to configure certain settings at runtime, on a per-job basis rather than globally within flink-conf.yaml? For example, I have a job where it's desirable to retain a large number of checkpoints via state.checkpoints.num-retained. The checkpoints are cheap, and it's low cost. For other jobs, I don't want such a large number.