Hi, all Thanks for the reply and feedback for everyone!
After combining everyone's comments, the main concerns, and corresponding adjustments are as follows. @Guowei Ma, Thanks for your feedback. > should we introduce a _new_ non-orthogonal option(`taskmanager.memory.network.required-buffer-per-gate.max`). That is to say, the option will affect both streaming and batch shuffle behavior at the same time. 1. Because the default option can meet most requirements no matter in Streaming or Batch scenarios. We do not want users to adjust this default config option by design. This configuration option is added only to preserve the possibility of modification options for users. 2. In a few cases, if you really want to adjust this option, users may not expect to adjust the option according to Streaming or Batch, for example, according to the parallelism of the job. 3. Regarding the performance of streaming shuffle, the same problem of insufficient memory also exists for Streaming jobs. We introduced this configuration to enable users to decouple memory and parallelism, but it will affect some performance. By default, the feature is disabled and does not affect performance. However, the added configuration enables users to choose to decouple memory usage and parallelism for Streaming jobs. > It's better not to expose more implementation-related concepts to users. Thanks for you suggestion. I will modify the option name to avoid exposing implementation-related concepts. I have changed it to `taskmanager.memory.network.read-required-buffer.max` in the FLIP. @Dong Lin, Thanks for your reply. > it might be helpful to add a dedicated public interface section to describe the config key and config semantics. Thanks for your suggestion. I have added public interface section to describe the config key and config semantics clearly. > This FLIP seems to add more configs without removing any config from Flink. This Flip is to reduce the number of options to be adjusted when using Flink. After the Flip, the default option can meet the requirements in most sceneries rather than modifying any config options(`taskmanager.network.memory.buffers-per-channel` and `taskmanager.network.memory.floating-buffers-per-gate`), which is helpful to improve the out-of-box usability. In the long run, these two parameters `taskmanager.network.memory.buffers-per-channel` and `taskmanager.network.memory.floating-buffers-per-gate` may indeed be deprecated to reduce user parameters, but from the perspective of compatibility, we need to pay attention to users' feedback before deciding to deprecate the options. @Yanfei Lei,Thanks for your feedback. 1. Through the option is cluster level, the default value is different according to the job type. In other words, by default, for Batch jobs, the config value is enabled, 1000. And for Streaming jobs, the config value is not enabled by default. 2. I think this is a good point. The total floating buffers will not change with ExclusiveBuffersPerChannel(taskmanager.network.memory.buffers-per-channel) because this is the maximum memory threshold. But if the user explicitly specified the ExclusiveBuffersPerChannel, the calculated result of ExclusiveBuffersPerChannel * numChannels will change with it. Thanks again for all feedback! Best, Yuxin Zhu Zhu <reed...@gmail.com> 于2022年12月26日周一 17:18写道： > Hi Yuxin, > > Thanks for creating this FLIP. > > It's good if Flink does not require users to set a very large network > memory, or tune the advanced(hard-to-understand) per-channel/per-gate > buffer configs, to avoid "Insufficient number of network buffers" > exceptions > which can easily happen for large scale jobs. > > Regarding the new config > "taskmanager.memory.network.read-required-buffer.max", > I think it's still an advanced config which users may feel hard to tune. > However, given that in most cases users will not need to set it, I > think it's acceptable. > > So +1 for this FLIP. > > In the future, I think Flink should adaptively select to use exclusive > buffers > or not according to whether there are sufficient network buffers at > runtime. > Users then no longer need to understand the above configuration. This may > require supporting transitions between exclusive buffers and floating > buffers. > A problem of all buffer floating is that too few network buffers can result > in task slowness which is hard to identify by users. So it's also needed to > do improvements on metrics and web UI to expose such issues. > > Thanks, > Zhu > > Yanfei Lei <fredia...@gmail.com> 于2022年12月26日周一 11:13写道： > > > > Hi Yuxin, > > > > Thanks for the proposal! > > > > After reading the FLIP, I have some questions about the default value. > > This FLIP seems to introduce a *new* config > > option(taskmanager.memory.network.required-buffer-per-gate.max) to > control > > the network memory usage. > > 1. Is this configuration at the job level or cluster level? As the FLIP > > described, the default values of the Batch job and Stream job are > > different, If an explicit value is set for cluster level, will it affect > > all Batch jobs and Stream jobs on the cluster? > > > > 2. The default value of Batch Job depends on the value of > > > ExclusiveBuffersPerChannel(taskmanager.network.memory.buffers-per-channel), > > if the value of ExclusiveBuffersPerChannel changed, does > > "taskmanager.memory.network.required-buffer-per-gate.max" need to change > > with it? > > > > > > Best, > > Yanfei > > > > Dong Lin <lindon...@gmail.com> 于2022年12月25日周日 08:58写道： > > > > > Hi Yuxin, > > > > > > Thanks for proposing the FLIP! > > > > > > The motivation section makes sense. But it seems that the proposed > change > > > section mixes the proposed config with the evaluation results. It is a > bit > > > hard to understand what configs are proposed and how to describe these > > > configs to users. Given that the configuration setting is part of > public > > > interfaces, it might be helpful to add a dedicated public interface > section > > > to describe the config key and config semantics, as suggested in the > FLIP > > > template here > > > < > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > > . > > > > > > This FLIP seems to add more configs without removing any config from > Flink. > > > Intuitively this can make the Flink configuration harder rather than > > > simpler. Maybe we can get a better idea after we add a public interface > > > section to clarify those configs. > > > > > > Thanks, > > > Dong > > > > > > > > > On Mon, Dec 19, 2022 at 3:36 PM Yuxin Tan <tanyuxinw...@gmail.com> > wrote: > > > > > > > Hi, devs, > > > > > > > > I'd like to start a discussion about FLIP-266: Simplify network > memory > > > > configurations for TaskManager[1]. > > > > > > > > When using Flink, users may encounter the following issues that > affect > > > > usability. > > > > 1. The job may fail with an "Insufficient number of network buffers" > > > > exception. > > > > 2. Flink network memory size adjustment is complex. > > > > When encountering these issues, users can solve some problems by > adding > > > or > > > > adjusting parameters. However, multiple memory config options should > be > > > > changed. The config option adjustment requires understanding the > detailed > > > > internal implementation, which is impractical for most users. > > > > > > > > To simplify network memory configurations for TaskManager and improve > > > Flink > > > > usability, this FLIP proposed some optimization solutions for the > issues. > > > > > > > > Looking forward to your feedback. > > > > > > > > [1] > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager > > > > > > > > Best regards, > > > > Yuxin > > > > > > > >
