Hi Wei,
I agree with Dawid that we should defer the instantiation of temporary
functions to compile time. In the long-term, we would like to integrate
FunctionCatalog as a component of CatalogManager and unify the handling
of catalog objects as much as possible.
We should aim for your proposed option 1. For fluent definition of
functions in Table API, we would still like to offer passing instances
like `t.select(call(new ScalarFunction() { ... }))` that would be
registered as temporary system functions.
Regrds,
Timo
On 09.03.20 09:24, Wei Zhong wrote:
Hi Dawid,
I think defering the instantiation of temporary functions to compile time is
quite a good idea but needs further discussion. As it is orthogonal with this
FLIP, we could continue the discussion in a new thread later. What do you think?
Best,
Wei
在 2020年3月5日，21:11，Wei Zhong <weizhong0...@gmail.com> 写道：
Hi Dawid,
Thanks for your suggestion.
After some investigation, there are two designs in my mind about how to defer
the instantiation of temporary system function and temporary catalog function
to compile time.
1. FunctionCatalog accepts both FunctionDefinitions and uninstantiated
temporary functions. The uninstantiated temporary functions will be
instantiated when compiling. There is no public API change in this design, but
the FunctionCatalog needs to store and process both FunctionDefinitions and
uninstantiated temporary functions.
2. FunctionCatalog accepts only uninstantiated temporary functions. In this
design we need to remove those APIs that accepts FunctionDefinitions from
TableEnvironment, i.e. `void createTemporaryFunction(String path,
UserDefinedFunction functionInstance)` and `void
createTemporarySystemFunction(String name, UserDefinedFunction
functionInstance)`. But the FunctionCatalog only needs to store and process
uninstantiated temporary functions.
As I don't know the details about the plan to store temporary functions as
catalog functions instead of FunctionDefinitions, I'm not sure which solution
fits more. It would be great if you could share more details or share some
thoughts on these two solutions?
Best,
Wei
在 2020年3月4日，16:17，Dawid Wysakowicz <dwysakow...@apache.org> 写道：
Hi all,
I had a really quick look and from my perspective the proposal looks fine.
I share Jarks opinion that the instantiation could be done at a later
stage. I agree with Wei it requires some changes in the internal
implementation of the FunctionCatalog, to store temporary functions as
catalog functions instead of FunctionDefinitions, but we have that on our
agenda anyway. I would suggest investigating if we could do that as part of
this flip already. Nevertheless this in theory can be also done later.
Best,
Dawid
On Mon, 2 Mar 2020, 14:58 Jark Wu, <imj...@gmail.com> wrote:
Thanks for the explanation, Wei!
On Mon, 2 Mar 2020 at 20:59, Wei Zhong <weizhong0...@gmail.com> wrote:
Hi Jark,
Thanks for your suggestion.
Actually, the timing of starting a Python process depends on the UDF
type,
because the Python process is used to provide the necessary information
to
instantiate the FunctionDefinition object of the Python UDF. For catalog
function, the FunctionDefinition will be instantiated when compiling the
job, which means the Python process is required during the compilation
instead of the registeration. For temporary system function and temporary
catalog function, the FunctionDefinition will be instantiated during the
UDF registeration, so the Python process need to be started at that time.
But this FLIP will only support registering the temporary system function
and temporary catalog function in SQL DDL because registering Python UDF
to
catalog is not supported yet. We plan to support the registeration of
Python catalog function (via Table API and SQL DDL) in a separate FLIP.
I'll add a non-goal section to the FLIP page to illustrate this.
Best,
Wei
在 2020年3月2日，15:11，Jark Wu <imj...@gmail.com> 写道：
Hi Weizhong,
Thanks for proposing this feature. In geneal, I'm +1 from the table's
view.
I have one suggestion: I think the register python function into
catalog
doesn't need to startup python process (the "High Level Sequence
Diagram"
in your FLIP).
Because only meta-information is persisted into catalog, we don't need
to
store "return type", "input types" into catalog.
I guess the python process is required when compiling a SQL job.
Best,
Jark
On Fri, 28 Feb 2020 at 19:04, Benchao Li <libenc...@gmail.com> wrote:
Big +1 for this feature.
We built our SQL platform on Java Table API, and most common UDF are
implemented in Java. However some python developers are not familiar
with
Java/Scala, and it's very inconvenient for these users to use UDF in
SQL.
Wei Zhong <weizhong0...@gmail.com> 于2020年2月28日周五 下午6:58写道：
Thank for your reply Dan!
By the way, this FLIP is closely related to the SQL API. @Jark Wu <
imj...@gmail.com> @Timo <twal...@apache.org> could you please take a
look?
Thanks,
Wei
在 2020年2月25日，16:25，zoudan <zoud...@163.com> 写道：
+1 for supporting Python UDF in Java/Scala Table API.
This is a great feature and would be helpful for python users!
Best,
Dan Zou
--
Benchao Li
School of Electronics Engineering and Computer Science, Peking
University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn
