@Gyula: I was thinking of the use case where the managed platform also handles
the job submission.
Setup #1
External user : Creating jar
------------------------------------
Managed platform : Submit job
Managed platform : Manage Flink cluster
As compared to
Setup #2
External user : Creating jar
External user : Submit job
------------------------------------
Managed platform : Manage Flink cluster
But I take your point that a REST API toggle doesn't make sense at all in Setup
#2, so it would be more inclusive to include the cluster configuration since
that works for both setups
Regards,
Hong
On 18/08/2022, 11:38, "Gyula Fóra" <gyula.f...@gmail.com> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know the
content is safe.
+1 for the proposal.
@Hong : I feel that the inverted override flag should be a cluster setting
and not something the user can override at will. I fear that this might
defeat the purpose of the feature itself.
So I think we should add a config for this but not expose it on the rest api
Gyula
On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <lian...@amazon.co.uk.invalid>
wrote:
> +1 to this FLIP.
> This is very useful for teams building a Flink platform to run jobs from
> an external user
>
> +1 on Danny's comment on adding a configuration to allow inverted order of
> overrides.
> However, might it be better to include an "override" toggle in the REST
> API itself? That way we can change the flink configuration override
> behaviour without restarting the Flink cluster. This would make sense if
we
> are thinking of a Session cluster and deploying multiple Flink jobs to the
> same cluster.
>
> Regards,
> Hong
>
>
> On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> +1 thanks for driving this FLIP.
>
> We actually have an internally forked equivalent of this, which is on
> our
> list to try to upstream. Your proposal would *almost* work "off the
> shelf"
> for us. We have an inverted order or priority:
> - Rest API / Flink CLI > User Code > Cluster Config
>
> This is because our end users do not submit their jobs directly, our
> service does it on their behalf. For our usecase we do not want to
> allow
> users to override certain values we set, since some are managed from
> our
> service configuration, example: checkpointing interval.
>
> How would you feel about including a cluster configuration to invert
> the
> order? This could be decoupled to a follow-up FLIP.
>
> Thanks,
> Danny Cranmer
>
>
> On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <jam.gz...@gmail.com>
> wrote:
>
> > This is a good suggestion, we can start this work after we finish
the
> > discussion and vote
> >
> > --
> > Best
> >
> > ConradJam
> >
> >
> > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道：
> >
> > > I've created a new ticket [FLINK-28973] Extending
> /jars/:jarid/plan API
> > to
> > > support setting Flink configs - ASF JIRA (apache.org)<
> > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> plan
> > API.
> > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > FLINK-27060
> > > & FLINK-28973 as subtasks. WDYT?
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: zhengyu chen <jam.gz...@gmail.com>
> > > Sent: Monday, August 15, 2022 18:06
> > > To: dev@flink.apache.org <dev@flink.apache.org>
> > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> Flink Rest
> > > Api
> > >
> > > Hi all,
> > >
> > >
> > > We would like to start a discussion thread on FLIP-256 Support Job
> > Dynamic
> > > Parameter With Flink Rest Api
> > > <
> > >
> >
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > [1]
> > >
> > > After the user submits the jar package, running a job through
> restapi
> > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > savepointPath, programArg, entry-class, parallelism) parameters,
> which is
> > > obvious with the diversification of job parameters (eg Checkpoint
> > address)
> > >
> > > This solves the problem that the user can pass in other parameters
> when
> > > submitting a job, avoiding the user to define these job parameters
> in the
> > > code, resulting in the need to repackage the job for each
> modification
> > >
> > > There was some interest from users [3] from a meetup and the
> mailing
> > list.
> > > Looking forward to comments and feedback, thanks!
> > >
> > >
> > > [1]
> > >
> > >
> >
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > >
> > >
> > > [2]
> > >
> > >
> >
>
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > >
> > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > >
> > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
>
>
