Thanks Junrui for the quick clarification! Sounds good to me, +1 for this FLIP.
Best, Rui On Thu, Oct 10, 2024 at 10:52 AM Junrui Lee <jrlee....@gmail.com> wrote: > Hi Rui, > > Thank you for your response. Here are my replies to your questions: > > Regarding question 1: We plan to label the config options on the Flink > configuration page in the upcoming 2.0 version. In the long term, our goal > is to ensure that all config options visible to users are properly labeled. > > Regarding question 2: That’s a good question, and we have indeed considered > similar approaches for pre-validating configuration items. However, due to > the current limitations of Flink, the ConfigOption acts as a format used > when retrieving config values from the Configuration object. Consequently, > the relationship between ConfigOptions and the key-value pairs stored in > the Configuration object is currently one-way; there is no way to reverse > lookup a ConfigOption from a string key. > > This implies that we might need to construct a complete collection of > ConfigOptions in advance, which is not an easy or straightforward task. > Therefore, we prefer to focus this FLIP on introducing the concept of > config option scope, while any potential functionality extensions can be > addressed in future FLIPs. > > WDYT? > > Best, > > Junrui > > Rui Fan <1996fan...@gmail.com> 于2024年10月10日周四 10:18写道： > > > Thanks Junrui for driving this proposal! > > > > It's valuable for users to distinguish all flink options. > > I have 2 minor questions about this proposal: > > > > 1. Does this FLIP enrich the scope for all old flink options? > > 2. If the option is configured at the wrong level, could we > > output some warn log to remind users? For example, > > an option of cluster level is added at job configuration. > > > > Best, > > Rui > > > > On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <decq12y...@gmail.com> wrote: > > > > > Thanks Junrui for starting this discussion! > > > Prior to this, it takes some experience to correctly distinguish > whether > > a > > > configuration item is effective at CLUSTER or JOB level, and this > > proposal > > > avoids such a burden. > > > And the FLIP looks good to me. > > > > > > > > > Best, > > > Yanquan > > > > > > Zhu Zhu <reed...@gmail.com> 于2024年10月9日周三 17:14写道： > > > > > > > Thanks Junrui for starting this discussion! > > > > +1 for the proposal > > > > > > > > Clarifying the scopes of config options can be beneficial for users. > > > > Even Flink developers occasionally encounter confusion regarding the > > > > scopes of various configurations, such as attempting to retrieve > > > > job-specific settings from `TaskExecutor#taskManagerConfiguration`. > > > > Clarifying the options' scopes in documentation can be a good > reference > > > > and reminder. > > > > > > > > Thanks, > > > > Zhu > > > > > > > > Junrui Lee <jrlee....@gmail.com> 于2024年10月9日周三 11:08写道： > > > > > > > > > Hi devs, > > > > > > > > > > I would like to initiate a discussion about FLIP-478: Introduce > > Config > > > > > Option Scope > > > > > > > > > > Currently, Flink can be configured in various ways. Taking a > session > > > job > > > > as > > > > > an example: > > > > > > > > > > 1. When the session cluster starts, the JobManager (JM) and > > TaskManager > > > > > (TM) load the configuration files config.yaml from specified > > directory > > > on > > > > > their respective machines, merging them with command line > parameters > > > (-D) > > > > > to form their component configuration. > > > > > > > > > > 2. When submitting a job, the client loads its local configuration > > file > > > > and > > > > > merges it with configuration within the job code and command line > > > > > parameters to obtain the job configuration. > > > > > > > > > > When this job is submitted to the session cluster, the values of > > > > > cluster-level config options, such as the memory for JM and TM, > will > > be > > > > > defined by the JM/TM configuration. And the values of job-level > > config > > > > > options, such as parallelism, will be defined by the job > > configuration. > > > > > Additionally, some config options will fall back to the values > > defined > > > in > > > > > the configuration from JM/TM if their values are not defined in job > > > > > configuration, such as restart strategies. > > > > > > > > > > However, it can be confusing for users to know where to set a > config > > > > > option. For example, we've received several bug reports from users > > > saying > > > > > that taskmanager.numberOfTaskSlots doesn’t take effect, which > turned > > > out > > > > to > > > > > be because they mistakenly configured it in the job code. > > > > > > > > > > As a result, we intend to introduce the concept of Config Option > > Scope > > > to > > > > > help users understand the proper way to configure each config > option. > > > And > > > > > the scopes should be displayed in Flink configuration > documentation. > > > > > > > > > > For more details, please refer to FLIP-478 [1]. Looking forward to > > your > > > > > feedback. > > > > > > > > > > Best, > > > > > > > > > > Junrui Lee > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope > > > > > > > > > > > > > > >
