Hi Zheng Yu, We would have to take the same "cluster configuration" (cannot be set on job submission) vs "job configuration" approach in the User code as well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster configuration. That way, neither the REST API / User code can override this configuration, and it can only be set in cluster configuration on startup.
There are other cluster specific configurations that are already treated this way (e.g. jobmanager.rpc.address, jobmanager.memory.process.size). As part of this work, I wonder if we can update the Flink docs on configuration (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/) to specify which configuration is "cluster" and which is "job". Regards, Hong On 20/08/2022, 09:16, "Zheng Yu Chen" <jam.gz...@gmail.com> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. @Gyula say that add a config for this but not expose it on the rest api is right when user start up session cluster,we can config (eg jobmanager.rest.api.submit.job.allow-reset-config = true/false) to controller this behavior Now we have an existing cluster, the admin configures checkpointing interval by this session cluster jobmanager.rest.api.submit.job.allow-reset-config = true/false - true : allow user reset new value config - false(default) : not allow user set new config from ,throw exp to tell user what properties is admin was set it However, there will be a problem in doing this. If the user writes this Flink Config in jar by hardcoding, the highest priority must be the code at this time, so the problem may still exist, and the user cannot be prevented from this behavior. what do you think ? @Hong @Gyula @Teoh @Danny Gyula Fóra <gyula.f...@gmail.com> 于2022年8月18日周四 18:37写道： > > +1 for the proposal. > > @Hong : I feel that the inverted override flag should be a cluster setting > and not something the user can override at will. I fear that this might > defeat the purpose of the feature itself. > So I think we should add a config for this but not expose it on the rest api > > Gyula > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <lian...@amazon.co.uk.invalid> > wrote: > > > +1 to this FLIP. > > This is very useful for teams building a Flink platform to run jobs from > > an external user > > > > +1 on Danny's comment on adding a configuration to allow inverted order of > > overrides. > > However, might it be better to include an "override" toggle in the REST > > API itself? That way we can change the flink configuration override > > behaviour without restarting the Flink cluster. This would make sense if we > > are thinking of a Session cluster and deploying multiple Flink jobs to the > > same cluster. > > > > Regards, > > Hong > > > > > > On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org> wrote: > > > > CAUTION: This email originated from outside of the organization. Do > > not click links or open attachments unless you can confirm the sender and > > know the content is safe. > > > > > > > > +1 thanks for driving this FLIP. > > > > We actually have an internally forked equivalent of this, which is on > > our > > list to try to upstream. Your proposal would *almost* work "off the > > shelf" > > for us. We have an inverted order or priority: > > - Rest API / Flink CLI > User Code > Cluster Config > > > > This is because our end users do not submit their jobs directly, our > > service does it on their behalf. For our usecase we do not want to > > allow > > users to override certain values we set, since some are managed from > > our > > service configuration, example: checkpointing interval. > > > > How would you feel about including a cluster configuration to invert > > the > > order? This could be decoupled to a follow-up FLIP. > > > > Thanks, > > Danny Cranmer > > > > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <jam.gz...@gmail.com> > > wrote: > > > > > This is a good suggestion, we can start this work after we finish the > > > discussion and vote > > > > > > -- > > > Best > > > > > > ConradJam > > > > > > > > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道： > > > > > > > I've created a new ticket [FLINK-28973] Extending > > /jars/:jarid/plan API > > > to > > > > support setting Flink configs - ASF JIRA (apache.org)< > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job > > plan > > > API. > > > > Maybe we can create a new umbrella ticket for FLIP-256 and put > > > FLINK-27060 > > > > & FLINK-28973 as subtasks. WDYT? > > > > > > > > Best, > > > > Zhanghao Chen > > > > ________________________________ > > > > From: zhengyu chen <jam.gz...@gmail.com> > > > > Sent: Monday, August 15, 2022 18:06 > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With > > Flink Rest > > > > Api > > > > > > > > Hi all, > > > > > > > > > > > > We would like to start a discussion thread on FLIP-256 Support Job > > > Dynamic > > > > Parameter With Flink Rest Api > > > > < > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > [1] > > > > > > > > After the user submits the jar package, running a job through > > restapi > > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState, > > > > savepointPath, programArg, entry-class, parallelism) parameters, > > which is > > > > obvious with the diversification of job parameters (eg Checkpoint > > > address) > > > > > > > > This solves the problem that the user can pass in other parameters > > when > > > > submitting a job, avoiding the user to define these job parameters > > in the > > > > code, resulting in the need to repackage the job for each > > modification > > > > > > > > There was some interest from users [3] from a meetup and the > > mailing > > > list. > > > > Looking forward to comments and feedback, thanks! > > > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > > > > [2] > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run > > > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060 > > > > > > > > > > > > > > > > > > > > -- > > > > Best > > > > > > > > ConradJam > > > > > > > > > > >
