Hi Timo, Yes, I think about convenient is a trade-off, for now think convenient compare with always do right thing I prefer add some limitation ensure user never encounter issues. So, #2 we are on the same page now.
Best, Jincheng Aljoscha Krettek <[email protected]> 于2019年9月4日周三 下午8:58写道： > Hi, > > Things looks interesting so far! > > I had one question: Where will most of the support code for this live? > Will this add the required code to flink-table-common or the different > runners? Can we implement this in such a way that only a minimal amount of > support code is required in the parts of the Table API (and Table API > runners) that are not python specific? > > Best, > Aljoscha > > > On 4. Sep 2019, at 14:14, Timo Walther <[email protected]> wrote: > > > > Hi Jincheng, > > > > 2. Serializability of functions: "#2 is very convenient for users" means > only until they have the first backwards-compatibility issue, after that > they will find it not so convinient anymore and will ask why the framework > allowed storing such objects in a persistent storage. I don't want to be > picky about it, but wanted to raise awareness that sometimes it is ok to > limit use cases to guide users for devloping backwards-compatible programs. > > > > Thanks for the explanation fo the remaining items. It sounds reasonable > to me. Regarding the example with `getKind()`, I actually meant > `org.apache.flink.table.functions.ScalarFunction#getKind` we don't allow > users to override this property. And I think we should do something similar > for the getLanguage property. > > > > Thanks, > > Timo > > > > On 03.09.19 15:01, jincheng sun wrote: > >> Hi Timo, > >> > >> Thanks for the quick reply ! :) > >> I have added more example for #3 and #5 to the FLIP. That are great > >> suggestions ! > >> > >> Regarding 2: > >> > >> There are two kind Serialization for CloudPickle(Which is different from > >> Java): > >> 1) For class and function which can be imported, CloudPickle only > >> serialize the full path of the class and function (just like java class > >> name). > >> 2) For the class and function which can not be imported, CloudPickle > will > >> serialize the full content of the class and function. > >> For #2, It means that we can not just store the full path of the class > and > >> function. > >> > >> The above serialization is recursive. > >> > >> However, there is indeed an problem of backwards compatibility when the > >> module path of the parent class changed. But I think this is an rare > case > >> and acceptable. i.e., For Flink framework we never change the user > >> interface module path if we want to keep backwards compatibility. For > user > >> code, if they change the interface of UDF's parent, they should > re-register > >> their functions. > >> > >> If we do not want support #2, we can store the full path of class and > >> function, in that case we have no backwards compatibility problem. But I > >> think the #2 is very convenient for users. > >> > >> What do you think? > >> > >> Regarding 4: > >> As I mentioned earlier, there may be built-in Python functions and I > think > >> language is a "function" concept. Function and Language are orthogonal > >> concepts. > >> We may have R, GO and other language functions in the future, not only > >> user-defined, but also built-in functions. > >> > >> You are right that users will not set this method and for Python > functions, > >> it will be set in the code-generated Java function by the framework. > So, I > >> think we should declare the getLanguage() in FunctionDefinition for now. > >> (I'm not pretty sure what do you mean by saying that getKind() is final > in > >> UserDefinedFunction?) > >> > >> Best, > >> Jincheng > >> > >> Timo Walther <[email protected]> 于2019年9月3日周二 下午6:01写道： > >> > >>> Hi Jincheng, > >>> > >>> thanks for your response. > >>> > >>> 2. Serializability of functions: Using some arbitrary serialization > >>> format for shipping a function to worker sounds fine to me. But once we > >>> store functions a the catalog we need to think about backwards > >>> compatibility and evolution of interfaces etc. I'm not sure if > >>> CloudPickle is the right long-term storage format for this. If we don't > >>> think about this in advance, we are basically violating our code > quality > >>> guide [1] of never use Java Serialization but in the Python-way. We are > >>> using the RPC serialization for persistence. > >>> > >>> 3. TableEnvironment: Can you add some example to the FLIP? Because API > >>> code like the following is not covered there: > >>> > >>> self.t_env.register_function("add_one", udf(lambda i: i + 1, > >>> DataTypes.BIGINT(), > >>> DataTypes.BIGINT())) > >>> self.t_env.register_function("subtract_one", udf(SubtractOne(), > >>> DataTypes.BIGINT(), > >>> DataTypes.BIGINT())) > >>> self.t_env.register_function("add", add) > >>> > >>> 4. FunctionDefinition: Your response still doesn't answer my question > >>> entirely. Why do we need FunctionDefinition.getLanguage() if this is a > >>> "user-defined function" concept and not a "function" concept. In any > >>> case, all users should not be able to set this method. So it must be > >>> final in UserDefinedFunction similar to getKind(). > >>> > >>> 5. Function characteristics: If UserDefinedFunction is defined in > >>> Python, why is it not used in your example in FLIP-58. You could you > >>> extend the example to show how to specify these attributes in the FLIP? > >>> > >>> Regards, > >>> Timo > >>> > >>> [1] > https://flink.apache.org/contributing/code-style-and-quality-java.html > >>> > >>> On 02.09.19 15:35, jincheng sun wrote: > >>>> Hi Timo, > >>>> > >>>> Great thanks for your feedback. I would like to share my thoughts with > >>> you > >>>> inline. :) > >>>> > >>>> Best, > >>>> Jincheng > >>>> > >>>> Timo Walther <[email protected]> 于2019年9月2日周一 下午5:04写道： > >>>> > >>>>> Hi all, > >>>>> > >>>>> the FLIP looks awesome. However, I would like to discuss the changes > to > >>>>> the user-facing parts again. Some feedback: > >>>>> > >>>>> 1. DataViews: With the current non-annotation design for DataViews, > we > >>>>> cannot perform eager state declaration, right? At which point during > >>>>> execution do we know which state is required by the function? We > need to > >>>>> instantiate the function first, right? > >>>>> > >>>>>> We will analysis the Python AggregateFunction and extract the > DataViews > >>>> used in the Python AggregateFunction. This can be done > >>>> by instantiate a Python AggregateFunction, creating an accumulator by > >>>> calling method create_accumulator and then analysis the created > >>>> accumulator. This is actually similar to the way that Java > >>>> AggregateFunction processing codegen logic. The extracted DataViews > can > >>>> then be used to construct the StateDescriptors in the operator, i.e., > we > >>>> should have hold the state spec and the state descriptor id in Java > >>>> operator and Python worker can access the state by specifying the > >>>> corresponding state descriptor id. > >>>> > >>>> > >>>> > >>>>> 2. Serializability of functions: How do we ensure serializability of > >>>>> functions for catalog persistence? In the Scala/Java API, we would > like > >>>>> to register classes instead of instances soon. This is the only way > to > >>>>> store a function properly in a catalog or we need some > >>>>> serialization/deserialization logic in the function interfaces to > >>>>> convert an instance to string properties. > >>>>> > >>>>>> The Python function will be serialized with CloudPickle anyway in > the > >>>> Python API as we need to transfer it to the Python worker which can > then > >>>> deserialize it for execution. The serialized Python function can be > >>> stored > >>>> into catalog. > >>>> > >>>> > >>>> > >>>>> 3. TableEnvironment: What is the signature of > `register_function(self, > >>>>> name, function)`? Does it accept both a class and function? Like > `class > >>>>> Sum` and `def split()`? Could you add some examples for registering > both > >>>>> kinds of functions? > >>>>> > >>>>>> It has been already supported which you mentioned. You can find an > >>>> example in the POC code: > >>>> > >>> > https://github.com/dianfu/flink/commit/93f41ba173482226af7513fdec5acba72b274489#diff-34f619b31a7e38604e22a42a441fbe2fR26 > >>>> > >>>> > >>>>> 4. FunctionDefinition: Function definition is not a user-defined > >>>>> function definition. It is the highest interface for both > user-defined > >>>>> and built-in functions. I'm not sure if getLanguage() should be part > of > >>>>> this interface or one-level down which would be > `UserDefinedFunction`. > >>>>> Built-in functions will never be implemented in a different > language. In > >>>>> any case, I would vote for removing the UNKNOWN language, because it > >>>>> does not solve anything. Why should a user declare a function that > the > >>>>> runtime can not handle? I also find the term `JAVA` confusing for > Scala > >>>>> users. How about `FunctionLanguage.JVM` instead? > >>>>> > >>>>>> Actually we may have built-in Python functions in the future. > Regarding > >>>> to the following expression: py_udf1(a, b) + py_udf2(c), if there is > >>>> built-in Python > >>>> funciton for '+' operator, then we don't need to mix using Java and > >>> Python > >>>> UDFs. In this way, we can improve the execution performance. > >>>> Regarding to removing FunctionLanguage.UNKNOWN and renaming > >>>> FunctionLanguage.Java to FunctionLanguage.JVM, it makes more sense to > me. > >>>> > >>>> > >>>> > >>>>> 5. Function characteristics: In the current design, function classes > do > >>>>> not extend from any upper class. How can users declare > characteristics > >>>>> that are present in `FunctionDefinition` like determinism, > requirements, > >>>>> or soon also monotonism. > >>>>> > >>>>>> Actually we have defined 'UserDefinedFunction' which is the base > class > >>>> for all user-defined functions. > >>>> We can define the deterministic, requirements, etc in this class. > >>>> Currently, we have already supported to define the deterministic. > >>>> > >>>> > >>>> > >>>>> Thanks, > >>>>> Timo > >>>>> > >>>>> > >>>>> On 02.09.19 03:38, Shaoxuan Wang wrote: > >>>>>> Hi Jincheng, Fudian, and Aljoscha, > >>>>>> I am assuming the proposed python UDX can also be applied to Flink > SQL. > >>>>>> Is this correct? If yes, I would suggest to title the FLIP as "Flink > >>>>> Python > >>>>>> User-Defined Function" or "Flink Python User-Defined Function for > >>> Table". > >>>>>> Regards, > >>>>>> Shaoxuan > >>>>>> > >>>>>> > >>>>>> On Wed, Aug 28, 2019 at 12:22 PM jincheng sun < > >>> [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Thanks for the feedback Bowen! > >>>>>>> > >>>>>>> Great thanks for create the FLIP and bring up the VOTE Dian! > >>>>>>> > >>>>>>> Best, Jincheng > >>>>>>> > >>>>>>> Dian Fu <[email protected]> 于2019年8月28日周三 上午11:32写道： > >>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> I have started a voting thread [1]. Thanks a lot for your help > during > >>>>>>>> creating the FLIP @Jincheng. > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi Bowen, > >>>>>>>> > >>>>>>>> Very appreciated for your comments. I have replied you in the > design > >>>>> doc. > >>>>>>>> As it seems that the comments doesn't affect the overall design, > I'll > >>>>> not > >>>>>>>> cancel the vote for now and we can continue the discussion in the > >>>>> design > >>>>>>>> doc. > >>>>>>>> > >>>>>>>> [1] > >>>>>>>> > >>> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-58-Flink-Python-User-Defined-Function-for-Table-API-td32295.html > >>>>>>>> < > >>>>>>>> > >>> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-58-Flink-Python-User-Defined-Function-for-Table-API-td32295.html > >>>>>>>> Regards, > >>>>>>>> Dian > >>>>>>>> > >>>>>>>>> 在 2019年8月28日，上午11:05，Bowen Li <[email protected]> 写道： > >>>>>>>>> > >>>>>>>>> Hi Jincheng and Dian, > >>>>>>>>> > >>>>>>>>> Sorry for being late to the party. I took a glance at the > proposal, > >>>>>>> LGTM > >>>>>>>> in > >>>>>>>>> general, and I left only a couple comments. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Bowen > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Aug 26, 2019 at 8:05 PM Dian Fu <[email protected]> > >>>>> wrote: > >>>>>>>>>> Hi Jincheng, > >>>>>>>>>> > >>>>>>>>>> Thanks! It works. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Dian > >>>>>>>>>> > >>>>>>>>>>> 在 2019年8月27日，上午10:55，jincheng sun <[email protected]> > 写道： > >>>>>>>>>>> > >>>>>>>>>>> Hi Dian, can you check if you have edit access? :) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Dian Fu <[email protected]> 于2019年8月26日周一 上午10:52写道： > >>>>>>>>>>> > >>>>>>>>>>>> Hi Jincheng, > >>>>>>>>>>>> > >>>>>>>>>>>> Appreciated for the kind tips and offering of help. Definitely > >>> need > >>>>>>>> it! > >>>>>>>>>>>> Could you grant me write permission for confluence? My Id: > Dian > >>> Fu > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Dian > >>>>>>>>>>>> > >>>>>>>>>>>>> 在 2019年8月26日，上午9:53，jincheng sun <[email protected]> > 写道： > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for your feedback Hequn & Dian. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Dian, I am glad to see that you want help to create the FLIP! > >>>>>>>>>>>>> Everyone will have first time, and I am very willing to help > you > >>>>>>>>>> complete > >>>>>>>>>>>>> your first FLIP creation. Here some tips: > >>>>>>>>>>>>> > >>>>>>>>>>>>> - First I'll give your account write permission for > confluence. > >>>>>>>>>>>>> - Before create the FLIP, please have look at the FLIP > Template > >>>>>>> [1], > >>>>>>>>>>>> (It's > >>>>>>>>>>>>> better to know more about FLIP by reading [2]) > >>>>>>>>>>>>> - Create Flink Python UDFs related JIRAs after completing the > >>> VOTE > >>>>>>> of > >>>>>>>>>>>>> FLIP.(I think you also can bring up the VOTE thread, if you > >>> want! > >>>>> ) > >>>>>>>>>>>>> Any problems you encounter during this period，feel free to > tell > >>> me > >>>>>>>> that > >>>>>>>>>>>> we > >>>>>>>>>>>>> can solve them together. :) > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Jincheng > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> [1] > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template > >>>>>>>>>>>>> [2] > >>>>>>>>>>>>> > >>> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > >>>>>>>>>>>>> Hequn Cheng <[email protected]> 于2019年8月23日周五 上午11:54写道： > >>>>>>>>>>>>> > >>>>>>>>>>>>>> +1 for starting the vote. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks Jincheng a lot for the discussion. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best, Hequn > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Aug 23, 2019 at 10:06 AM Dian Fu < > >>> [email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>> Hi Jincheng, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> +1 to start the FLIP create and VOTE on this feature. I'm > >>>>> willing > >>>>>>>> to > >>>>>>>>>>>> help > >>>>>>>>>>>>>>> on the FLIP create if you don't mind. As I haven't created > a > >>>>> FLIP > >>>>>>>>>>>> before, > >>>>>>>>>>>>>>> it will be great if you could help on this. :) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>> Dian > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 在 2019年8月22日，下午11:41，jincheng sun < > [email protected]> > >>>>>>> 写道： > >>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks a lot for your feedback. If there are no more > >>>>> suggestions > >>>>>>>> and > >>>>>>>>>>>>>>>> comments, I think it's better to initiate a vote to > create a > >>>>>>> FLIP > >>>>>>>>>> for > >>>>>>>>>>>>>>>> Apache Flink Python UDFs. > >>>>>>>>>>>>>>>> What do you think? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best, Jincheng > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> jincheng sun <[email protected]> 于2019年8月15日周四 > >>>>>>> 上午12:54写道： > >>>>>>>>>>>>>>>>> Hi Thomas, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks for your confirmation and the very important > reminder > >>>>>>>> about > >>>>>>>>>>>>>>> bundle > >>>>>>>>>>>>>>>>> processing. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have had add the description about how to perform > bundle > >>>>>>>>>> processing > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>> the perspective of checkpoint and watermark. Feel free to > >>>>> leave > >>>>>>>>>>>>>>> comments if > >>>>>>>>>>>>>>>>> there are anything not describe clearly. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>> Jincheng > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Dian Fu <[email protected]> 于2019年8月14日周三 上午10:08写道： > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi Thomas, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks a lot the suggestions. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Regarding to bundle processing, there is a section > >>>>>>>> "Checkpoint"[1] > >>>>>>>>>>>> in > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> design doc which talks about how to handle the > checkpoint. > >>>>>>>>>>>>>>>>>> However, I think you are right that we should talk more > >>> about > >>>>>>>> it, > >>>>>>>>>>>>>> such > >>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>> what's bundle processing, how it affects the checkpoint > and > >>>>>>>>>>>>>> watermark, > >>>>>>>>>>>>>>> how > >>>>>>>>>>>>>>>>>> to handle the checkpoint and watermark, etc. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>> > >>> > https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3 > >>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>> > >>> > https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3 > >>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>> Dian > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 在 2019年8月14日，上午1:01，Thomas Weise <[email protected]> 写道： > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi Jincheng, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks for putting this together. The proposal is very > >>>>>>>> detailed, > >>>>>>>>>>>>>>>>>> thorough > >>>>>>>>>>>>>>>>>>> and for me as a Beam Flink runner contributor easy to > >>>>>>>> understand > >>>>>>>>>> :) > >>>>>>>>>>>>>>>>>>> One thing that you should probably detail more is the > >>> bundle > >>>>>>>>>>>>>>>>>> processing. It > >>>>>>>>>>>>>>>>>>> is critically important for performance that multiple > >>>>>>> elements > >>>>>>>>>> are > >>>>>>>>>>>>>>>>>>> processed in a bundle. The default bundle size in the > >>> Flink > >>>>>>>>>> runner > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>> 1s or > >>>>>>>>>>>>>>>>>>> 1000 elements, whichever comes first. And for > streaming, > >>> you > >>>>>>>> can > >>>>>>>>>>>>>> find > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> logic necessary to align the bundle processing with > >>>>>>> watermarks > >>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> checkpointing here: > >>>>>>>>>>>>>>>>>>> > >>> > https://github.com/apache/beam/blob/release-2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java > >>>>>>>>>>>>>>>>>>> Thomas > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Tue, Aug 13, 2019 at 7:05 AM jincheng sun < > >>>>>>>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The Python Table API(without Python UDF support) has > >>>>> already > >>>>>>>>>> been > >>>>>>>>>>>>>>>>>> supported > >>>>>>>>>>>>>>>>>>>> and will be available in the coming release 1.9. > >>>>>>>>>>>>>>>>>>>> As Python UDF is very important for Python users, we'd > >>> like > >>>>>>> to > >>>>>>>>>>>>>> start > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> discussion about the Python UDF support in the Python > >>> Table > >>>>>>>> API. > >>>>>>>>>>>>>>>>>>>> Aljoscha Krettek, Dian Fu and I have discussed offline > >>> and > >>>>>>>> have > >>>>>>>>>>>>>>>>>> drafted a > >>>>>>>>>>>>>>>>>>>> design doc[1]. It includes the following items: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> - The user-defined function interfaces. > >>>>>>>>>>>>>>>>>>>> - The user-defined function execution architecture. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> As mentioned by many guys in the previous discussion > >>>>>>>> thread[2], > >>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>> portability framework was introduced in Apache Beam in > >>>>>>> latest > >>>>>>>>>>>>>>>>>> releases. It > >>>>>>>>>>>>>>>>>>>> provides well-defined, language-neutral data > structures > >>> and > >>>>>>>>>>>>>> protocols > >>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>> language-neutral user-defined function execution. This > >>>>>>> design > >>>>>>>> is > >>>>>>>>>>>>>>> based > >>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>> Beam's portability framework. We will introduce how to > >>> make > >>>>>>>> use > >>>>>>>>>> of > >>>>>>>>>>>>>>>>>> Beam's > >>>>>>>>>>>>>>>>>>>> portability framework for user-defined function > >>> execution: > >>>>>>>> data > >>>>>>>>>>>>>>>>>>>> transmission, state access, checkpoint, metrics, > logging, > >>>>>>> etc. > >>>>>>>>>>>>>>>>>>>> Considering that the design relies on Beam's > portability > >>>>>>>>>> framework > >>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>> Python user-defined function execution and not all the > >>>>>>>>>>>> contributors > >>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>> Flink community are familiar with Beam's portability > >>>>>>>> framework, > >>>>>>>>>> we > >>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>> done a prototype[3] for proof of concept and also > ease of > >>>>>>>>>>>>>>>>>> understanding of > >>>>>>>>>>>>>>>>>>>> the design. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Welcome any feedback. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>> Jincheng > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>> > https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit?usp=sharing > >>>>>>>>>>>>>>>>>>>> [2] > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html > >>>>>>>>>>>>>>>>>>>> [3] https://github.com/dianfu/flink/commits/udf_poc > >>>>>>>>>>>>>>>>>>>> > >>> > > > >
