Hi Gyula, FLINK-33548 proposes adding a new resource field to match with Kubernetes pod resource configuration. Here's my suggestion: instead of adding a new resource field, let's use a pod template for more advanced resource setup. Adding a new resource field might confuse users. This change can also help with issues when users use Flink Kubernetes commands directly, without the operator.
Thanks Surendra On Tue, Dec 5, 2023 at 3:10 PM richard.su <richardsuc...@gmail.com> wrote: > Sorry Gyula, let me explain more about the point of 2, if I avoid the > override, I will got a jobmanager pod still with resources consist with > “jobmanager.memory.process.size”, but a flinkdeployment with a resource > larger than that. > > Thanks for your time. > Richard Su > > > 2023年12月5日 17:13，richard.su <richardsuc...@gmail.com> 写道： > > > > Thank you for your time, Gyula, I have more question about Flink-33548, > we can have more discussion about this and make progress: > > > > 1. I agree with you about declaring resources in FlinkDeployment > resource sections. But Flink Operator will override the > “jobmanager.memory.process.size” and "taskmanager.memory.process.size", > despite I have set these configuration or not in flink configuration. If > user had configured all memory attributes, the override will leads to error > as the overall computation is error. > > > > the code of override is in FlinkConfigManager.class in buildFrom method, > which apply to JobmanagerSpec and TaskManagerSpec. > > > > 2. If I modified the code of override, I will still encounter this issue > of FLINK-24150, because I only modified the code of flink operator but not > flink-kubernetes package, so I will make a pod resources like (cpu:1c > memory:1g) and container resource to be (cpu:1c, memory 850m), because I > already set jobmanager.memory.process.size to 850m. > > > > 3. because of there two point, we need to make the podTemplate have > higher priority. Otherwise we can refactor the code of flink operator, > which should import something new configuration to support the native mode. > > > > I think it will be better to import some configuration, which > FlinkConfigManager.class can override it using the resource of > JobmanagerSpec and TaskManagerSpec. > > > > When it deep into the code flink-kubernetes package, we using these new > configuration as the final result of containers resources. > > > > Thanks for your time. > > Richard Su > > > >> 2023年12月5日 16:45，Gyula Fóra <gyula.f...@gmail.com> 写道： > >> > >> As you can see in the jira ticket there hasn't been any progress, nobody > >> started to work on this yet. > >> > >> I personally don't think it's confusing to declare resources in the > >> FlinkDeployment resource sections. It's well documented and worked very > >> well so far for most users. > >> This is pretty common practice for kubernetes. > >> > >> Cheers, > >> Gyula > >> > >> On Tue, Dec 5, 2023 at 9:35 AM richard.su <richardsuc...@gmail.com> > wrote: > >> > >>> Hi, Gyula, is there had any progress in FLINK-33548? I would like to > join > >>> the discussion but I haven't seen any discussion in the url. > >>> > >>> I also make flinkdeployment by flink operator, which indeed will > override > >>> the process size by TaskmanagerSpec.resources or > JobmanagerSpec.resources, > >>> which really confused, I had modified the code of flink operator to > avoid > >>> the override. > >>> > >>> Looking for your response. > >>> > >>> Thank you. > >>> Richard Su > >>> > >>> > >>>> 2023年12月5日 16:22，Gyula Fóra <gyula.f...@gmail.com> 写道： > >>>> > >>>> Hi! > >>>> > >>>> Please see the discussion in > >>>> https://lists.apache.org/thread/6p5tk6obmk1qxf169so498z4vk8cg969 > >>>> and the ticket: https://issues.apache.org/jira/browse/FLINK-33548 > >>>> > >>>> We should follow the approach outlined there. If you are interested > you > >>> are > >>>> welcome to pick up the operator ticket. > >>>> > >>>> Unfortunately your PR can be a large unexpected change to existing > users > >>> so > >>>> we should not add it. > >>>> > >>>> Cheers, > >>>> Gyula > >>>> > >>>> On Tue, Dec 5, 2023 at 9:05 AM 苏超腾 <richardsuc...@gmail.com> wrote: > >>>> > >>>>> Hello everyone, > >>>>> > >>>>> I've encountered an issue while using flink kubernetes native, > Despite > >>>>> setting resource limits in the pod template, it appears that these > >>> limits > >>>>> and requests are not considered during JobManager(JM) and TaskManager > >>> (TM) > >>>>> pod deployment. > >>>>> > >>>>> I find the a issue had opened in jira FLINK-24150, which introduced > >>>>> almost the same questions that I encountered. > >>>>> > >>>>> I agrees that if user had provided pod templates, we should put > priority > >>>>> on it higher than flink calculated from configuration. > >>>>> > >>>>> But this need some discussion in our community, because it related > some > >>>>> scenarios: > >>>>> If I want to create a pod with Graranted QoS and want the memory of > the > >>>>> Flink main container to be larger than the process size of Flink, I > >>> cannot > >>>>> directly modify podTemplate (although we can use limit factor, this > will > >>>>> cause the QoS to change from Graranted to Burstable) > >>>>> If I want to create a pod with Burstable QoS, I don't want to use > limit > >>>>> actor and want to directly configure the request to be 50% of the > limit, > >>>>> which cannot be modified. > >>>>> In order to meet these scenarios, I had committed a pull request > >>>>> https://github.com/apache/flink/pull/23872 > >>>>> > >>>>> This code is very simple and just need someone to review, this pr > can be > >>>>> cherry pick to other old version, which will be helpful. > >>>>> > >>>>> > >>>>> I would appreciate any feedback on this. > >>>>> > >>>>> Thank you for your time and contributions to the Flink project. > >>>>> > >>>>> Thank you, > >>>>> chaoran.su > >>> > >>> > > > >
