There's 2 kinds of configuration: job level & cluster level. I am afraid we don't have document to differentiate that, it depends on how user understand these configuration. We may need to improve document on that.
Kurt Young <ykt...@gmail.com> 于2020年3月6日周五 上午8:34写道： > If you already have a running flink cluster and you want submit another > job to this cluster, then all the configurations > relates to process parameters like TM memory, slot number etc are not be > able to modify. > > Best, > Kurt > > > On Thu, Mar 5, 2020 at 11:08 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > >> Kurt can you please explain which conf parameters do you mean? >> >> In regular executions (Yarn for instance) we have dynamic config >> parameters overriding any flink-conf argument. >> So it is not about setting them in the user code but it should happen >> before the ClusterDescriptors are created (ie in the together with the >> CustomCommandLine logic) >> >> Gyula >> >> On Thu, Mar 5, 2020 at 3:49 PM Kurt Young <ykt...@gmail.com> wrote: >> >>> IIRC the tricky thing here is not all the config options belong to >>> flink-conf.yaml can be adjust dynamically in user's program. >>> So it will end up like some of the configurations can be overridden but >>> some are not. The experience is not quite good for users. >>> >>> Best, >>> Kurt >>> >>> >>> On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zjf...@gmail.com> wrote: >>> >>>> Hi Gyula, >>>> >>>> I am doing integration Flink with Zeppelin. One feature in Zeppelin is >>>> that user could override any features in flink-conf.yaml. (Actually any >>>> features here >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html). >>>> Of course you can run flink sql in Zeppelin, and could also leverage other >>>> features of Zeppelin, like visualization. >>>> >>>> If you are interested, you could try the master branch of Zeppelin + >>>> this improvement PR >>>> >>>> https://github.com/apache/zeppelin >>>> https://github.com/apache/zeppelin/pull/3676 >>>> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md >>>> >>>> >>>> >>>> >>>> >>>> >>>> Gyula Fóra <gyula.f...@gmail.com> 于2020年3月5日周四 下午6:51写道： >>>> >>>>> I could basically list a few things I want to set (execution.target >>>>> for example), but it's fair to assume that I would like to be able to set >>>>> anything :) >>>>> >>>>> Gyula >>>>> >>>>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <jingsongl...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Gyula, >>>>>> >>>>>> Maybe Blink planner has invoked >>>>>> "StreamExecutionEnvironment.configure", which planner do you use? >>>>>> >>>>>> But "StreamExecutionEnvironment.configure" is only for partial >>>>>> configuration, can not for all configuration in flink-conf.yaml. >>>>>> So what's the config do you want to set? I know some config like >>>>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not >>>>>> set.... >>>>>> >>>>>> Best, >>>>>> Jingsong Lee >>>>>> >>>>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <imj...@gmail.com> wrote: >>>>>> >>>>>>> Hi Gyula, >>>>>>> >>>>>>> Flink configurations can be overrided via >>>>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set >>>>>>> Table >>>>>>> specific configs. >>>>>>> I will think it as a bug/improvement of SQL CLI which should be >>>>>>> fixed in 1.10.1. >>>>>>> >>>>>>> Best, >>>>>>> Jark >>>>>>> >>>>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gyula.f...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Caizhi, >>>>>>>> >>>>>>>> This seems like a pretty big shortcoming for any >>>>>>>> multi-user/multi-app environment. I will open a jira for this. >>>>>>>> >>>>>>>> Gyula >>>>>>>> >>>>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <tsreape...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Gyula. >>>>>>>>> >>>>>>>>> I'm afraid there is no way to override all Flink configurations >>>>>>>>> currently. SQL client yaml file can only override some of the Flink >>>>>>>>> configurations. >>>>>>>>> >>>>>>>>> Configuration entries indeed can only set Table specific configs, >>>>>>>>> while deployment entires are used to set the result fetching address >>>>>>>>> and >>>>>>>>> port. There is currently no way to change the execution target from >>>>>>>>> the SQL >>>>>>>>> client. >>>>>>>>> >>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> 于2020年3月5日周四 下午4:31写道： >>>>>>>>> >>>>>>>>>> Hi All! >>>>>>>>>> >>>>>>>>>> I am trying to understand if there is any way to override flink >>>>>>>>>> configuration parameters when starting the SQL Client. >>>>>>>>>> >>>>>>>>>> It seems that the only way to pass any parameters is through the >>>>>>>>>> environment yaml. >>>>>>>>>> >>>>>>>>>> There I found 2 possible routes: >>>>>>>>>> >>>>>>>>>> configuration: this doesn't work as it only sets Table specific >>>>>>>>>> configs apparently, but maybe I am wrong. >>>>>>>>>> >>>>>>>>>> deployment: I tried using dynamic properties options here but >>>>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is >>>>>>>>>> impossible to >>>>>>>>>> pass options like -yD or -D. >>>>>>>>>> >>>>>>>>>> Does anyone have any suggestions? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Gyula >>>>>>>>>> >>>>>>>>> >>>>>> >>>>>> -- >>>>>> Best, Jingsong Lee >>>>>> >>>>> >>>> >>>> -- >>>> Best Regards >>>> >>>> Jeff Zhang >>>> >>> -- Best Regards Jeff Zhang
