Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi David, Thanks for the feedback! > For the proposed deprecations, will there be a proposed recommended > alternatives that will be mentioned in the deprecation. If they are going to > be removed for v2, it would be good to explicitly document the thinking and > impact of the deprecations and consequent removals, Some options to be deprecated are considered too complicated for user to tweak. There will be alternatives for those deprecated options, e.g., the option related to the floating buffer. While some options have a reasonable default value and there are hardly tweaked, those options will not have an alternative and the current default value will be used, e.g., the options related to the netty. And we will document the reason for the deprecation in 1.20. Best, Xuannan On Thu, May 16, 2024 at 12:01 AM David Radley wrote: > > Hi Xuannan, > I like that you are cleaning up options that I assume are not recommended or > currently used in some way. > > I have not got experience of these options. For the proposed deprecations, > will there be a proposed recommended alternatives that will be mentioned in > the deprecation. If they are going to be removed for v2, it would be good to > explicitly document the thinking and impact of the deprecations and > consequent removals, > Kind regards, David. > > From: Xuannan Su > Date: Tuesday, 14 May 2024 at 02:08 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration > for Flink 2.0 > Hi all, > > Thank you for all the comments and suggestions! If there are no > further comments, I'd like to close the discussion and start the > voting in two days. > > Best regards, > Xuannan > > > > On Mon, May 13, 2024 at 3:10 PM Jeyhun Karimov wrote: > > > > Hi Xuannan, > > > > Thanks a lot for the update. The FLIP looks good to me. +1 for it. > > > > Regards, > > Jeyhun > > > > On Mon, May 13, 2024 at 4:45 AM Xuannan Su wrote: > > > > > Hi Jeyhun, > > > > > > Thanks for the comment! > > > > > > Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated > > > the FLIP to include the information. > > > > > > Best regards, > > > Xuannan > > > > > > On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov > > > wrote: > > > > > > > > Hi Xuannan, > > > > > > > > Thanks for driving this FLIP! > > > > I have a minor comment. Do we plan to remove StreamPipelineOptions in > > > 2.0, > > > > as it only contains deprecated options? > > > > > > > > Regards, > > > > Jeyhun > > > > > > > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > > > Thanks Xuannan for the update! > > > > > > > > > > LGTM, +1 for this proposal. > > > > > > > > > > Best, > > > > > Rui > > > > > > > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su > > > wrote: > > > > > > > > > > > Hi Rui, > > > > > > > > > > > > Thanks for the suggestion! > > > > > > > > > > > > I updated the description of > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > > > > hard-coded it to 20. > > > > > > > > > > > > Best regards, > > > > > > Xuannan > > > > > > > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> > > > wrote: > > > > > > > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will > > > > > > > > be > > > > > > removed > > > > > > > and hard-coded to either 10 or 20. > > > > > > > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > > > > the overdraft buffer in the current FLIP? > > > > > > > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > > > > - Removing this option means users cannot change it, it might be > > > better > > > > > > to > > > > > > > turn it up. > > > > >
RE: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Xuannan, I like that you are cleaning up options that I assume are not recommended or currently used in some way. I have not got experience of these options. For the proposed deprecations, will there be a proposed recommended alternatives that will be mentioned in the deprecation. If they are going to be removed for v2, it would be good to explicitly document the thinking and impact of the deprecations and consequent removals, Kind regards, David. From: Xuannan Su Date: Tuesday, 14 May 2024 at 02:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0 Hi all, Thank you for all the comments and suggestions! If there are no further comments, I'd like to close the discussion and start the voting in two days. Best regards, Xuannan On Mon, May 13, 2024 at 3:10 PM Jeyhun Karimov wrote: > > Hi Xuannan, > > Thanks a lot for the update. The FLIP looks good to me. +1 for it. > > Regards, > Jeyhun > > On Mon, May 13, 2024 at 4:45 AM Xuannan Su wrote: > > > Hi Jeyhun, > > > > Thanks for the comment! > > > > Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated > > the FLIP to include the information. > > > > Best regards, > > Xuannan > > > > On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov > > wrote: > > > > > > Hi Xuannan, > > > > > > Thanks for driving this FLIP! > > > I have a minor comment. Do we plan to remove StreamPipelineOptions in > > 2.0, > > > as it only contains deprecated options? > > > > > > Regards, > > > Jeyhun > > > > > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > Thanks Xuannan for the update! > > > > > > > > LGTM, +1 for this proposal. > > > > > > > > Best, > > > > Rui > > > > > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su > > wrote: > > > > > > > > > Hi Rui, > > > > > > > > > > Thanks for the suggestion! > > > > > > > > > > I updated the description of > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > > > hard-coded it to 20. > > > > > > > > > > Best regards, > > > > > Xuannan > > > > > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> > > wrote: > > > > > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > > > > removed > > > > > > and hard-coded to either 10 or 20. > > > > > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > > > the overdraft buffer in the current FLIP? > > > > > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > > > - Removing this option means users cannot change it, it might be > > better > > > > > to > > > > > > turn it up. > > > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > > > doesn't > > > > > > introduce more risk. > > > > > > > > > > > > Best, > > > > > > Rui > > > > > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > > > wrote: > > > > > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > > > > > +1 for the proposal. > > > > > > > > > > > > > > Best, > > > > > > > Yuxin > > > > > > > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > > > solely > > > > > > > > changing the configuration options, but are bound to changing / > > > > > removal > > > > > > > of > &
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi all, Thank you for all the comments and suggestions! If there are no further comments, I'd like to close the discussion and start the voting in two days. Best regards, Xuannan On Mon, May 13, 2024 at 3:10 PM Jeyhun Karimov wrote: > > Hi Xuannan, > > Thanks a lot for the update. The FLIP looks good to me. +1 for it. > > Regards, > Jeyhun > > On Mon, May 13, 2024 at 4:45 AM Xuannan Su wrote: > > > Hi Jeyhun, > > > > Thanks for the comment! > > > > Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated > > the FLIP to include the information. > > > > Best regards, > > Xuannan > > > > On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov > > wrote: > > > > > > Hi Xuannan, > > > > > > Thanks for driving this FLIP! > > > I have a minor comment. Do we plan to remove StreamPipelineOptions in > > 2.0, > > > as it only contains deprecated options? > > > > > > Regards, > > > Jeyhun > > > > > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > Thanks Xuannan for the update! > > > > > > > > LGTM, +1 for this proposal. > > > > > > > > Best, > > > > Rui > > > > > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su > > wrote: > > > > > > > > > Hi Rui, > > > > > > > > > > Thanks for the suggestion! > > > > > > > > > > I updated the description of > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > > > hard-coded it to 20. > > > > > > > > > > Best regards, > > > > > Xuannan > > > > > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> > > wrote: > > > > > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > > > > removed > > > > > > and hard-coded to either 10 or 20. > > > > > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > > > the overdraft buffer in the current FLIP? > > > > > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > > > - Removing this option means users cannot change it, it might be > > better > > > > > to > > > > > > turn it up. > > > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > > > doesn't > > > > > > introduce more risk. > > > > > > > > > > > > Best, > > > > > > Rui > > > > > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > > > wrote: > > > > > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > > > > > +1 for the proposal. > > > > > > > > > > > > > > Best, > > > > > > > Yuxin > > > > > > > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > > > solely > > > > > > > > changing the configuration options, but are bound to changing / > > > > > removal > > > > > > > of > > > > > > > > certain features. E.g., the removal of hash-blocking shuffle > > and > > > > > legacy > > > > > > > > hybrid shuffle mode, and the behavior change of overdraft > > network > > > > > > > buffers. > > > > > > > > Therefore, it might be nicer to provide an implementation plan > > > > with a > > > > > > > list > > > > > > > > of related tasks in the FLIP. This should not block the FLIP > > > > though. > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su < > > suxuanna...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, > > we > > > > have > > > > > > > > > revisited all runtime configurations and identified several > > > > > > > > > improvements to enhance user-friendliness and > > maintainability. In > > > > > this > > > > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > > > > you! > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Xuannan > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Xuannan, Thanks a lot for the update. The FLIP looks good to me. +1 for it. Regards, Jeyhun On Mon, May 13, 2024 at 4:45 AM Xuannan Su wrote: > Hi Jeyhun, > > Thanks for the comment! > > Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated > the FLIP to include the information. > > Best regards, > Xuannan > > On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov > wrote: > > > > Hi Xuannan, > > > > Thanks for driving this FLIP! > > I have a minor comment. Do we plan to remove StreamPipelineOptions in > 2.0, > > as it only contains deprecated options? > > > > Regards, > > Jeyhun > > > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > Thanks Xuannan for the update! > > > > > > LGTM, +1 for this proposal. > > > > > > Best, > > > Rui > > > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su > wrote: > > > > > > > Hi Rui, > > > > > > > > Thanks for the suggestion! > > > > > > > > I updated the description of > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > > hard-coded it to 20. > > > > > > > > Best regards, > > > > Xuannan > > > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> > wrote: > > > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > > > removed > > > > > and hard-coded to either 10 or 20. > > > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > > the overdraft buffer in the current FLIP? > > > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > > - Removing this option means users cannot change it, it might be > better > > > > to > > > > > turn it up. > > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > > doesn't > > > > > introduce more risk. > > > > > > > > > > Best, > > > > > Rui > > > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > > wrote: > > > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > > > +1 for the proposal. > > > > > > > > > > > > Best, > > > > > > Yuxin > > > > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > > solely > > > > > > > changing the configuration options, but are bound to changing / > > > > removal > > > > > > of > > > > > > > certain features. E.g., the removal of hash-blocking shuffle > and > > > > legacy > > > > > > > hybrid shuffle mode, and the behavior change of overdraft > network > > > > > > buffers. > > > > > > > Therefore, it might be nicer to provide an implementation plan > > > with a > > > > > > list > > > > > > > of related tasks in the FLIP. This should not block the FLIP > > > though. > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su < > suxuanna...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, > we > > > have > > > > > > > > revisited all runtime configurations and identified several > > > > > > > > improvements to enhance user-friendliness and > maintainability. In > > > > this > > > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > > > you! > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Xuannan > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Jeyhun, Thanks for the comment! Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated the FLIP to include the information. Best regards, Xuannan On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov wrote: > > Hi Xuannan, > > Thanks for driving this FLIP! > I have a minor comment. Do we plan to remove StreamPipelineOptions in 2.0, > as it only contains deprecated options? > > Regards, > Jeyhun > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > Thanks Xuannan for the update! > > > > LGTM, +1 for this proposal. > > > > Best, > > Rui > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su wrote: > > > > > Hi Rui, > > > > > > Thanks for the suggestion! > > > > > > I updated the description of > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > hard-coded it to 20. > > > > > > Best regards, > > > Xuannan > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > > removed > > > > and hard-coded to either 10 or 20. > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > the overdraft buffer in the current FLIP? > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > - Removing this option means users cannot change it, it might be better > > > to > > > > turn it up. > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > doesn't > > > > introduce more risk. > > > > > > > > Best, > > > > Rui > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > wrote: > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > +1 for the proposal. > > > > > > > > > > Best, > > > > > Yuxin > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > solely > > > > > > changing the configuration options, but are bound to changing / > > > removal > > > > > of > > > > > > certain features. E.g., the removal of hash-blocking shuffle and > > > legacy > > > > > > hybrid shuffle mode, and the behavior change of overdraft network > > > > > buffers. > > > > > > Therefore, it might be nicer to provide an implementation plan > > with a > > > > > list > > > > > > of related tasks in the FLIP. This should not block the FLIP > > though. > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we > > have > > > > > > > revisited all runtime configurations and identified several > > > > > > > improvements to enhance user-friendliness and maintainability. In > > > this > > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > > you! > > > > > > > > > > > > > > Best regards, > > > > > > > Xuannan > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Xuannan, Thanks for driving this FLIP! I have a minor comment. Do we plan to remove StreamPipelineOptions in 2.0, as it only contains deprecated options? Regards, Jeyhun On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > Thanks Xuannan for the update! > > LGTM, +1 for this proposal. > > Best, > Rui > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su wrote: > > > Hi Rui, > > > > Thanks for the suggestion! > > > > I updated the description of > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > hard-coded it to 20. > > > > Best regards, > > Xuannan > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > removed > > > and hard-coded to either 10 or 20. > > > > > > Currently, it's a public option. Could we determine the value of > > > the overdraft buffer in the current FLIP? > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > - Removing this option means users cannot change it, it might be better > > to > > > turn it up. > > > - Most of tasks don't use the overdraft buffer, so increasing it > doesn't > > > introduce more risk. > > > > > > Best, > > > Rui > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > wrote: > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > +1 for the proposal. > > > > > > > > Best, > > > > Yuxin > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > solely > > > > > changing the configuration options, but are bound to changing / > > removal > > > > of > > > > > certain features. E.g., the removal of hash-blocking shuffle and > > legacy > > > > > hybrid shuffle mode, and the behavior change of overdraft network > > > > buffers. > > > > > Therefore, it might be nicer to provide an implementation plan > with a > > > > list > > > > > of related tasks in the FLIP. This should not block the FLIP > though. > > > > > > > > > > Best, > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we > have > > > > > > revisited all runtime configurations and identified several > > > > > > improvements to enhance user-friendliness and maintainability. In > > this > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > you! > > > > > > > > > > > > Best regards, > > > > > > Xuannan > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Thanks Xuannan for the update! LGTM, +1 for this proposal. Best, Rui On Sat, May 11, 2024 at 10:20 AM Xuannan Su wrote: > Hi Rui, > > Thanks for the suggestion! > > I updated the description of > taskmanager.network.memory.max-overdraft-buffers-per-gate and > hard-coded it to 20. > > Best regards, > Xuannan > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > Thanks Xuannan for driving this proposal! > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > removed > > and hard-coded to either 10 or 20. > > > > Currently, it's a public option. Could we determine the value of > > the overdraft buffer in the current FLIP? > > > > I vote 20 as the hard code value due to 2 reasons: > > - Removing this option means users cannot change it, it might be better > to > > turn it up. > > - Most of tasks don't use the overdraft buffer, so increasing it doesn't > > introduce more risk. > > > > Best, > > Rui > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > wrote: > > > > > Thanks for the effort, Xuannan. > > > > > > +1 for the proposal. > > > > > > Best, > > > Yuxin > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > +1 for the proposed changes. > > > > > > > > Just one suggestion: Some of the proposed changes involve not solely > > > > changing the configuration options, but are bound to changing / > removal > > > of > > > > certain features. E.g., the removal of hash-blocking shuffle and > legacy > > > > hybrid shuffle mode, and the behavior change of overdraft network > > > buffers. > > > > Therefore, it might be nicer to provide an implementation plan with a > > > list > > > > of related tasks in the FLIP. This should not block the FLIP though. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > > > > > revisited all runtime configurations and identified several > > > > > improvements to enhance user-friendliness and maintainability. In > this > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > > > > > > > Best regards, > > > > > Xuannan > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Rui, Thanks for the suggestion! I updated the description of taskmanager.network.memory.max-overdraft-buffers-per-gate and hard-coded it to 20. Best regards, Xuannan On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote: > > Thanks Xuannan for driving this proposal! > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be removed > and hard-coded to either 10 or 20. > > Currently, it's a public option. Could we determine the value of > the overdraft buffer in the current FLIP? > > I vote 20 as the hard code value due to 2 reasons: > - Removing this option means users cannot change it, it might be better to > turn it up. > - Most of tasks don't use the overdraft buffer, so increasing it doesn't > introduce more risk. > > Best, > Rui > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan wrote: > > > Thanks for the effort, Xuannan. > > > > +1 for the proposal. > > > > Best, > > Yuxin > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > Thanks for driving this effort, Xuannan. > > > > > > +1 for the proposed changes. > > > > > > Just one suggestion: Some of the proposed changes involve not solely > > > changing the configuration options, but are bound to changing / removal > > of > > > certain features. E.g., the removal of hash-blocking shuffle and legacy > > > hybrid shuffle mode, and the behavior change of overdraft network > > buffers. > > > Therefore, it might be nicer to provide an implementation plan with a > > list > > > of related tasks in the FLIP. This should not block the FLIP though. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > > > > revisited all runtime configurations and identified several > > > > improvements to enhance user-friendliness and maintainability. In this > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > > > > > Best regards, > > > > Xuannan > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Thanks Xuannan for driving this! +1 for this proposal. Best, Junrui Rui Fan <1996fan...@gmail.com> 于2024年5月6日周一 11:27写道： > Thanks Xuannan for driving this proposal! > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be removed > and hard-coded to either 10 or 20. > > Currently, it's a public option. Could we determine the value of > the overdraft buffer in the current FLIP? > > I vote 20 as the hard code value due to 2 reasons: > - Removing this option means users cannot change it, it might be better to > turn it up. > - Most of tasks don't use the overdraft buffer, so increasing it doesn't > introduce more risk. > > Best, > Rui > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan wrote: > > > Thanks for the effort, Xuannan. > > > > +1 for the proposal. > > > > Best, > > Yuxin > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > Thanks for driving this effort, Xuannan. > > > > > > +1 for the proposed changes. > > > > > > Just one suggestion: Some of the proposed changes involve not solely > > > changing the configuration options, but are bound to changing / removal > > of > > > certain features. E.g., the removal of hash-blocking shuffle and legacy > > > hybrid shuffle mode, and the behavior change of overdraft network > > buffers. > > > Therefore, it might be nicer to provide an implementation plan with a > > list > > > of related tasks in the FLIP. This should not block the FLIP though. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > > > > revisited all runtime configurations and identified several > > > > improvements to enhance user-friendliness and maintainability. In > this > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > > > > > Best regards, > > > > Xuannan > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Xintong, Thanks for the suggestion! I updated the FLIP to include a list of related tasks. Best regards, Xuannan On Mon, Apr 29, 2024 at 3:40 PM Xintong Song wrote: > > Thanks for driving this effort, Xuannan. > > +1 for the proposed changes. > > Just one suggestion: Some of the proposed changes involve not solely > changing the configuration options, but are bound to changing / removal of > certain features. E.g., the removal of hash-blocking shuffle and legacy > hybrid shuffle mode, and the behavior change of overdraft network buffers. > Therefore, it might be nicer to provide an implementation plan with a list > of related tasks in the FLIP. This should not block the FLIP though. > > Best, > > Xintong > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su wrote: > > > Hi all, > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > > revisited all runtime configurations and identified several > > improvements to enhance user-friendliness and maintainability. In this > > FLIP, we aim to refine the runtime configuration. > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > Best regards, > > Xuannan > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Thanks Xuannan for driving this proposal! > taskmanager.network.memory.max-overdraft-buffers-per-gate will be removed and hard-coded to either 10 or 20. Currently, it's a public option. Could we determine the value of the overdraft buffer in the current FLIP? I vote 20 as the hard code value due to 2 reasons: - Removing this option means users cannot change it, it might be better to turn it up. - Most of tasks don't use the overdraft buffer, so increasing it doesn't introduce more risk. Best, Rui On Mon, May 6, 2024 at 10:47 AM Yuxin Tan wrote: > Thanks for the effort, Xuannan. > > +1 for the proposal. > > Best, > Yuxin > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > Thanks for driving this effort, Xuannan. > > > > +1 for the proposed changes. > > > > Just one suggestion: Some of the proposed changes involve not solely > > changing the configuration options, but are bound to changing / removal > of > > certain features. E.g., the removal of hash-blocking shuffle and legacy > > hybrid shuffle mode, and the behavior change of overdraft network > buffers. > > Therefore, it might be nicer to provide an implementation plan with a > list > > of related tasks in the FLIP. This should not block the FLIP though. > > > > Best, > > > > Xintong > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > wrote: > > > > > Hi all, > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > > > revisited all runtime configurations and identified several > > > improvements to enhance user-friendliness and maintainability. In this > > > FLIP, we aim to refine the runtime configuration. > > > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > > > Best regards, > > > Xuannan > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Thanks for the effort, Xuannan. +1 for the proposal. Best, Yuxin Xintong Song 于2024年4月29日周一 15:40写道： > Thanks for driving this effort, Xuannan. > > +1 for the proposed changes. > > Just one suggestion: Some of the proposed changes involve not solely > changing the configuration options, but are bound to changing / removal of > certain features. E.g., the removal of hash-blocking shuffle and legacy > hybrid shuffle mode, and the behavior change of overdraft network buffers. > Therefore, it might be nicer to provide an implementation plan with a list > of related tasks in the FLIP. This should not block the FLIP though. > > Best, > > Xintong > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su wrote: > > > Hi all, > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > > revisited all runtime configurations and identified several > > improvements to enhance user-friendliness and maintainability. In this > > FLIP, we aim to refine the runtime configuration. > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > Best regards, > > Xuannan > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Thanks for driving this effort, Xuannan. +1 for the proposed changes. Just one suggestion: Some of the proposed changes involve not solely changing the configuration options, but are bound to changing / removal of certain features. E.g., the removal of hash-blocking shuffle and legacy hybrid shuffle mode, and the behavior change of overdraft network buffers. Therefore, it might be nicer to provide an implementation plan with a list of related tasks in the FLIP. This should not block the FLIP though. Best, Xintong On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su wrote: > Hi all, > > I'd like to start a discussion on FLIP-450: Improve Runtime > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have > revisited all runtime configurations and identified several > improvements to enhance user-friendliness and maintainability. In this > FLIP, we aim to refine the runtime configuration. > > Looking forward to everyone's feedback and suggestions. Thank you! > > Best regards, > Xuannan > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 >
