Hi,
Things looks interesting so far!
I had one question: Where will most of the support code for this live? Will
this add the required code to flink-table-common or the different runners? Can
we implement this in such a way that only a minimal amount of support code is
required in the parts of the Table API (and Table API runners) that are not
python specific?
Best,
Aljoscha
> On 4. Sep 2019, at 14:14, Timo Walther <twal...@apache.org> wrote:
>
> Hi Jincheng,
>
> 2. Serializability of functions: "#2 is very convenient for users" means only
> until they have the first backwards-compatibility issue, after that they will
> find it not so convinient anymore and will ask why the framework allowed
> storing such objects in a persistent storage. I don't want to be picky about
> it, but wanted to raise awareness that sometimes it is ok to limit use cases
> to guide users for devloping backwards-compatible programs.
>
> Thanks for the explanation fo the remaining items. It sounds reasonable to
> me. Regarding the example with `getKind()`, I actually meant
> `org.apache.flink.table.functions.ScalarFunction#getKind` we don't allow
> users to override this property. And I think we should do something similar
> for the getLanguage property.
>
> Thanks,
> Timo
>
> On 03.09.19 15:01, jincheng sun wrote:
>> Hi Timo,
>>
>> Thanks for the quick reply ! :)
>> I have added more example for #3 and #5 to the FLIP. That are great
>> suggestions !
>>
>> Regarding 2:
>>
>> There are two kind Serialization for CloudPickle(Which is different from
>> Java):
>> 1) For class and function which can be imported, CloudPickle only
>> serialize the full path of the class and function (just like java class
>> name).
>> 2) For the class and function which can not be imported, CloudPickle will
>> serialize the full content of the class and function.
>> For #2, It means that we can not just store the full path of the class and
>> function.
>>
>> The above serialization is recursive.
>>
>> However, there is indeed an problem of backwards compatibility when the
>> module path of the parent class changed. But I think this is an rare case
>> and acceptable. i.e., For Flink framework we never change the user
>> interface module path if we want to keep backwards compatibility. For user
>> code, if they change the interface of UDF's parent, they should re-register
>> their functions.
>>
>> If we do not want support #2, we can store the full path of class and
>> function, in that case we have no backwards compatibility problem. But I
>> think the #2 is very convenient for users.
>>
>> What do you think?
>>
>> Regarding 4:
>> As I mentioned earlier, there may be built-in Python functions and I think
>> language is a "function" concept. Function and Language are orthogonal
>> concepts.
>> We may have R, GO and other language functions in the future, not only
>> user-defined, but also built-in functions.
>>
>> You are right that users will not set this method and for Python functions,
>> it will be set in the code-generated Java function by the framework. So, I
>> think we should declare the getLanguage() in FunctionDefinition for now.
>> (I'm not pretty sure what do you mean by saying that getKind() is final in
>> UserDefinedFunction?)
>>
>> Best,
>> Jincheng
>>
>> Timo Walther <twal...@apache.org> 于2019年9月3日周二 下午6:01写道：
>>
>>> Hi Jincheng,
>>>
>>> thanks for your response.
>>>
>>> 2. Serializability of functions: Using some arbitrary serialization
>>> format for shipping a function to worker sounds fine to me. But once we
>>> store functions a the catalog we need to think about backwards
>>> compatibility and evolution of interfaces etc. I'm not sure if
>>> CloudPickle is the right long-term storage format for this. If we don't
>>> think about this in advance, we are basically violating our code quality
>>> guide [1] of never use Java Serialization but in the Python-way. We are
>>> using the RPC serialization for persistence.
>>>
>>> 3. TableEnvironment: Can you add some example to the FLIP? Because API
>>> code like the following is not covered there:
>>>
>>> self.t_env.register_function("add_one", udf(lambda i: i + 1,
>>> DataTypes.BIGINT(),
>>> DataTypes.BIGINT()))
>>> self.t_env.register_function("subtract_one", udf(SubtractOne(),
>>> DataTypes.BIGINT(),
>>> DataTypes.BIGINT()))
>>> self.t_env.register_function("add", add)
>>>
>>> 4. FunctionDefinition: Your response still doesn't answer my question
>>> entirely. Why do we need FunctionDefinition.getLanguage() if this is a
>>> "user-defined function" concept and not a "function" concept. In any
>>> case, all users should not be able to set this method. So it must be
>>> final in UserDefinedFunction similar to getKind().
>>>
>>> 5. Function characteristics: If UserDefinedFunction is defined in
>>> Python, why is it not used in your example in FLIP-58. You could you
>>> extend the example to show how to specify these attributes in the FLIP?
>>>
>>> Regards,
>>> Timo
>>>
>>> [1] https://flink.apache.org/contributing/code-style-and-quality-java.html
>>>
>>> On 02.09.19 15:35, jincheng sun wrote:
>>>> Hi Timo,
>>>>
>>>> Great thanks for your feedback. I would like to share my thoughts with
>>> you
>>>> inline. :)
>>>>
>>>> Best,
>>>> Jincheng
>>>>
>>>> Timo Walther <twal...@apache.org> 于2019年9月2日周一 下午5:04写道：
>>>>
>>>>> Hi all,
>>>>>
>>>>> the FLIP looks awesome. However, I would like to discuss the changes to
>>>>> the user-facing parts again. Some feedback:
>>>>>
>>>>> 1. DataViews: With the current non-annotation design for DataViews, we
>>>>> cannot perform eager state declaration, right? At which point during
>>>>> execution do we know which state is required by the function? We need to
>>>>> instantiate the function first, right?
>>>>>
>>>>>> We will analysis the Python AggregateFunction and extract the DataViews
>>>> used in the Python AggregateFunction. This can be done
>>>> by instantiate a Python AggregateFunction, creating an accumulator by
>>>> calling method create_accumulator and then analysis the created
>>>> accumulator. This is actually similar to the way that Java
>>>> AggregateFunction processing codegen logic. The extracted DataViews can
>>>> then be used to construct the StateDescriptors in the operator, i.e., we
>>>> should have hold the state spec and the state descriptor id in Java
>>>> operator and Python worker can access the state by specifying the
>>>> corresponding state descriptor id.
>>>>
>>>>
>>>>
>>>>> 2. Serializability of functions: How do we ensure serializability of
>>>>> functions for catalog persistence? In the Scala/Java API, we would like
>>>>> to register classes instead of instances soon. This is the only way to
>>>>> store a function properly in a catalog or we need some
>>>>> serialization/deserialization logic in the function interfaces to
>>>>> convert an instance to string properties.
>>>>>
>>>>>> The Python function will be serialized with CloudPickle anyway in the
>>>> Python API as we need to transfer it to the Python worker which can then
>>>> deserialize it for execution. The serialized Python function can be
>>> stored
>>>> into catalog.
>>>>
>>>>
>>>>
>>>>> 3. TableEnvironment: What is the signature of `register_function(self,
>>>>> name, function)`? Does it accept both a class and function? Like `class
>>>>> Sum` and `def split()`? Could you add some examples for registering both
>>>>> kinds of functions?
>>>>>
>>>>>> It has been already supported which you mentioned. You can find an
>>>> example in the POC code:
>>>>
>>> https://github.com/dianfu/flink/commit/93f41ba173482226af7513fdec5acba72b274489#diff-34f619b31a7e38604e22a42a441fbe2fR26
>>>>
>>>>
>>>>> 4. FunctionDefinition: Function definition is not a user-defined
>>>>> function definition. It is the highest interface for both user-defined
>>>>> and built-in functions. I'm not sure if getLanguage() should be part of
>>>>> this interface or one-level down which would be `UserDefinedFunction`.
>>>>> Built-in functions will never be implemented in a different language. In
>>>>> any case, I would vote for removing the UNKNOWN language, because it
>>>>> does not solve anything. Why should a user declare a function that the
>>>>> runtime can not handle? I also find the term `JAVA` confusing for Scala
>>>>> users. How about `FunctionLanguage.JVM` instead?
>>>>>
>>>>>> Actually we may have built-in Python functions in the future. Regarding
>>>> to the following expression: py_udf1(a, b) + py_udf2(c), if there is
>>>> built-in Python
>>>> funciton for '+' operator, then we don't need to mix using Java and
>>> Python
>>>> UDFs. In this way, we can improve the execution performance.
>>>> Regarding to removing FunctionLanguage.UNKNOWN and renaming
>>>> FunctionLanguage.Java to FunctionLanguage.JVM, it makes more sense to me.
>>>>
>>>>
>>>>
>>>>> 5. Function characteristics: In the current design, function classes do
>>>>> not extend from any upper class. How can users declare characteristics
>>>>> that are present in `FunctionDefinition` like determinism, requirements,
>>>>> or soon also monotonism.
>>>>>
>>>>>> Actually we have defined 'UserDefinedFunction' which is the base class
>>>> for all user-defined functions.
>>>> We can define the deterministic, requirements, etc in this class.
>>>> Currently, we have already supported to define the deterministic.
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>> Timo
>>>>>
>>>>>
>>>>> On 02.09.19 03:38, Shaoxuan Wang wrote:
>>>>>> Hi Jincheng, Fudian, and Aljoscha,
>>>>>> I am assuming the proposed python UDX can also be applied to Flink SQL.
>>>>>> Is this correct? If yes, I would suggest to title the FLIP as "Flink
>>>>> Python
>>>>>> User-Defined Function" or "Flink Python User-Defined Function for
>>> Table".
>>>>>> Regards,
>>>>>> Shaoxuan
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 28, 2019 at 12:22 PM jincheng sun <
>>> sunjincheng...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the feedback Bowen!
>>>>>>>
>>>>>>> Great thanks for create the FLIP and bring up the VOTE Dian!
>>>>>>>
>>>>>>> Best, Jincheng
>>>>>>>
>>>>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月28日周三 上午11:32写道：
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I have started a voting thread [1]. Thanks a lot for your help during
>>>>>>>> creating the FLIP @Jincheng.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Bowen,
>>>>>>>>
>>>>>>>> Very appreciated for your comments. I have replied you in the design
>>>>> doc.
>>>>>>>> As it seems that the comments doesn't affect the overall design, I'll
>>>>> not
>>>>>>>> cancel the vote for now and we can continue the discussion in the
>>>>> design
>>>>>>>> doc.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-58-Flink-Python-User-Defined-Function-for-Table-API-td32295.html
>>>>>>>> <
>>>>>>>>
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-58-Flink-Python-User-Defined-Function-for-Table-API-td32295.html
>>>>>>>> Regards,
>>>>>>>> Dian
>>>>>>>>
>>>>>>>>> 在 2019年8月28日，上午11:05，Bowen Li <bowenl...@gmail.com> 写道：
>>>>>>>>>
>>>>>>>>> Hi Jincheng and Dian,
>>>>>>>>>
>>>>>>>>> Sorry for being late to the party. I took a glance at the proposal,
>>>>>>> LGTM
>>>>>>>> in
>>>>>>>>> general, and I left only a couple comments.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bowen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 26, 2019 at 8:05 PM Dian Fu <dian0511...@gmail.com>
>>>>> wrote:
>>>>>>>>>> Hi Jincheng,
>>>>>>>>>>
>>>>>>>>>> Thanks! It works.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Dian
>>>>>>>>>>
>>>>>>>>>>> 在 2019年8月27日，上午10:55，jincheng sun <sunjincheng...@gmail.com> 写道：
>>>>>>>>>>>
>>>>>>>>>>> Hi Dian, can you check if you have edit access? :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月26日周一 上午10:52写道：
>>>>>>>>>>>
>>>>>>>>>>>> Hi Jincheng,
>>>>>>>>>>>>
>>>>>>>>>>>> Appreciated for the kind tips and offering of help. Definitely
>>> need
>>>>>>>> it!
>>>>>>>>>>>> Could you grant me write permission for confluence? My Id: Dian
>>> Fu
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Dian
>>>>>>>>>>>>
>>>>>>>>>>>>> 在 2019年8月26日，上午9:53，jincheng sun <sunjincheng...@gmail.com> 写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your feedback Hequn & Dian.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dian, I am glad to see that you want help to create the FLIP!
>>>>>>>>>>>>> Everyone will have first time, and I am very willing to help you
>>>>>>>>>> complete
>>>>>>>>>>>>> your first FLIP creation. Here some tips:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - First I'll give your account write permission for confluence.
>>>>>>>>>>>>> - Before create the FLIP, please have look at the FLIP Template
>>>>>>> [1],
>>>>>>>>>>>> (It's
>>>>>>>>>>>>> better to know more about FLIP by reading [2])
>>>>>>>>>>>>> - Create Flink Python UDFs related JIRAs after completing the
>>> VOTE
>>>>>>> of
>>>>>>>>>>>>> FLIP.(I think you also can bring up the VOTE thread, if you
>>> want!
>>>>> )
>>>>>>>>>>>>> Any problems you encounter during this period，feel free to tell
>>> me
>>>>>>>> that
>>>>>>>>>>>> we
>>>>>>>>>>>>> can solve them together. :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Jincheng
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template
>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>>>>>>>>>>>> Hequn Cheng <chenghe...@gmail.com> 于2019年8月23日周五 上午11:54写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1 for starting the vote.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Jincheng a lot for the discussion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best, Hequn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 23, 2019 at 10:06 AM Dian Fu <
>>> dian0511...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Hi Jincheng,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +1 to start the FLIP create and VOTE on this feature. I'm
>>>>> willing
>>>>>>>> to
>>>>>>>>>>>> help
>>>>>>>>>>>>>>> on the FLIP create if you don't mind. As I haven't created a
>>>>> FLIP
>>>>>>>>>>>> before,
>>>>>>>>>>>>>>> it will be great if you could help on this. :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Dian
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 在 2019年8月22日，下午11:41，jincheng sun <sunjincheng...@gmail.com>
>>>>>>> 写道：
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks a lot for your feedback. If there are no more
>>>>> suggestions
>>>>>>>> and
>>>>>>>>>>>>>>>> comments, I think it's better to initiate a vote to create a
>>>>>>> FLIP
>>>>>>>>>> for
>>>>>>>>>>>>>>>> Apache Flink Python UDFs.
>>>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best, Jincheng
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> jincheng sun <sunjincheng...@gmail.com> 于2019年8月15日周四
>>>>>>> 上午12:54写道：
>>>>>>>>>>>>>>>>> Hi Thomas,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for your confirmation and the very important reminder
>>>>>>>> about
>>>>>>>>>>>>>>> bundle
>>>>>>>>>>>>>>>>> processing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have had add the description about how to perform bundle
>>>>>>>>>> processing
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> the perspective of checkpoint and watermark. Feel free to
>>>>> leave
>>>>>>>>>>>>>>> comments if
>>>>>>>>>>>>>>>>> there are anything not describe clearly.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Jincheng
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月14日周三 上午10:08写道：
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Thomas,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks a lot the suggestions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding to bundle processing, there is a section
>>>>>>>> "Checkpoint"[1]
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> design doc which talks about how to handle the checkpoint.
>>>>>>>>>>>>>>>>>> However, I think you are right that we should talk more
>>> about
>>>>>>>> it,
>>>>>>>>>>>>>> such
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> what's bundle processing, how it affects the checkpoint and
>>>>>>>>>>>>>> watermark,
>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> to handle the checkpoint and watermark, etc.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>
>>> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3
>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>
>>> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Dian
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 在 2019年8月14日，上午1:01，Thomas Weise <t...@apache.org> 写道：
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Jincheng,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for putting this together. The proposal is very
>>>>>>>> detailed,
>>>>>>>>>>>>>>>>>> thorough
>>>>>>>>>>>>>>>>>>> and for me as a Beam Flink runner contributor easy to
>>>>>>>> understand
>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>> One thing that you should probably detail more is the
>>> bundle
>>>>>>>>>>>>>>>>>> processing. It
>>>>>>>>>>>>>>>>>>> is critically important for performance that multiple
>>>>>>> elements
>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> processed in a bundle. The default bundle size in the
>>> Flink
>>>>>>>>>> runner
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> 1s or
>>>>>>>>>>>>>>>>>>> 1000 elements, whichever comes first. And for streaming,
>>> you
>>>>>>>> can
>>>>>>>>>>>>>> find
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> logic necessary to align the bundle processing with
>>>>>>> watermarks
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> checkpointing here:
>>>>>>>>>>>>>>>>>>>
>>> https://github.com/apache/beam/blob/release-2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
>>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Aug 13, 2019 at 7:05 AM jincheng sun <
>>>>>>>>>>>>>>> sunjincheng...@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The Python Table API(without Python UDF support) has
>>>>> already
>>>>>>>>>> been
>>>>>>>>>>>>>>>>>> supported
>>>>>>>>>>>>>>>>>>>> and will be available in the coming release 1.9.
>>>>>>>>>>>>>>>>>>>> As Python UDF is very important for Python users, we'd
>>> like
>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> discussion about the Python UDF support in the Python
>>> Table
>>>>>>>> API.
>>>>>>>>>>>>>>>>>>>> Aljoscha Krettek, Dian Fu and I have discussed offline
>>> and
>>>>>>>> have
>>>>>>>>>>>>>>>>>> drafted a
>>>>>>>>>>>>>>>>>>>> design doc[1]. It includes the following items:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - The user-defined function interfaces.
>>>>>>>>>>>>>>>>>>>> - The user-defined function execution architecture.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> As mentioned by many guys in the previous discussion
>>>>>>>> thread[2],
>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> portability framework was introduced in Apache Beam in
>>>>>>> latest
>>>>>>>>>>>>>>>>>> releases. It
>>>>>>>>>>>>>>>>>>>> provides well-defined, language-neutral data structures
>>> and
>>>>>>>>>>>>>> protocols
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> language-neutral user-defined function execution. This
>>>>>>> design
>>>>>>>> is
>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> Beam's portability framework. We will introduce how to
>>> make
>>>>>>>> use
>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> Beam's
>>>>>>>>>>>>>>>>>>>> portability framework for user-defined function
>>> execution:
>>>>>>>> data
>>>>>>>>>>>>>>>>>>>> transmission, state access, checkpoint, metrics, logging,
>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>> Considering that the design relies on Beam's portability
>>>>>>>>>> framework
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> Python user-defined function execution and not all the
>>>>>>>>>>>> contributors
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> Flink community are familiar with Beam's portability
>>>>>>>> framework,
>>>>>>>>>> we
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> done a prototype[3] for proof of concept and also ease of
>>>>>>>>>>>>>>>>>> understanding of
>>>>>>>>>>>>>>>>>>>> the design.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Welcome any feedback.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Jincheng
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
>>>>>>>>>>>>>>>>>>>> [3] https://github.com/dianfu/flink/commits/udf_poc
>>>>>>>>>>>>>>>>>>>>
>>>
>
