Thanks for driving this discussion, Dian! +1 for this proposal. It will help to reduce container failure due to the memory overuse. Some comments left in the design doc.
Best, Yangze Guo On Mon, Nov 18, 2019 at 4:06 PM Xintong Song <tonysong...@gmail.com> wrote: > > Sorry for the late reply. > > +1 for the general proposal. > > And one remainder, to use UNKNOWN resource requirement, we need to make > sure optimizer knowns which operators use off-heap managed memory, and > compute and set a fraction to the operators. See FLIP-53[1] for more > details, and I would suggest you to double check with @Zhu Zhu who works on > this part. > > Thank you~ > > Xintong Song > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > > On Tue, Nov 12, 2019 at 11:53 AM Dian Fu <dian0511...@gmail.com> wrote: > > > Hi Jincheng, > > > > Thanks for the reply and also looking forward to the feedback from the > > community. > > > > Thanks, > > Dian > > > > > 在 2019年11月11日，下午2:34，jincheng sun <sunjincheng...@gmail.com> 写道： > > > > > > Hi all, > > > > > > +1， Thanks for bring up this discussion Dian! > > > > > > The Resource Management is very important for PyFlink UDF. So, It's great > > > if anyone can add more comments or inputs in the design doc or feedback > > in > > > ML. :) > > > > > > Best, > > > Jincheng > > > > > > Dian Fu <dian0511...@gmail.com> 于2019年11月5日周二 上午11:32写道： > > > > > >> Hi everyone, > > >> > > >> In FLIP-58[1] it will add the support of Python user-defined stateless > > >> function for Python Table API. It will launch a separate Python process > > for > > >> Python user-defined function execution. The resources used by the Python > > >> process should be managed properly by Flink’s resource management > > >> framework. FLIP-49[2] has proposed a unified memory management framework > > >> and PyFlink user-defined function resource management should be based on > > >> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about > > this. I > > >> draft a design doc[3] and want to start a discussion about PyFlink > > >> user-defined function resource management. > > >> > > >> Welcome any comments on the design doc or giving us feedback on the ML > > >> directly. > > >> > > >> Regards, > > >> Dian > > >> > > >> [1] > > >> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > > >> [2] > > >> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > >> [3] > > >> > > https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m > > > >
