Re: The reason why I prefer option 3 is that in option 3 all objects internally are identified with 3 parts.
True, but the problem we have is not about how to differentiate each type objects internally. Rather, it's rather about how a user referencing an object unambiguously and consistently. Thanks, Xuefu On Wed, Sep 18, 2019 at 4:55 PM Dawid Wysakowicz <wysakowicz.da...@gmail.com> wrote: > Last additional comment on Option 2. The reason why I prefer option 3 is > that in option 3 all objects internally are identified with 3 parts. This > makes it easier to handle at different locations e.g. while persisting > views, as all objects have uniform representation. > > On Thu, 19 Sep 2019, 07:31 Dawid Wysakowicz, <wysakowicz.da...@gmail.com> > wrote: > > > Hi, > > I think it makes sense to start voting at this point. > > > > Option 1: Only 1-part identifiers > > PROS: > > - allows shadowing built-in functions > > CONS: > > - incosistent with all the other objects, both permanent & temporary > > - does not allow shadowing catalog functions > > > > Option 2: Special keyword for built-in function > > I think this is quite similar to the special catalog/db. The thing I am > > strongly against in this proposal is the GLOBAL keyword. This keyword > has a > > meaning in rdbms systems and means a function that is present for a > > lifetime of a session in which it was created, but available in all other > > sessions. Therefore I really don't want to use this keyword in a > different > > context. > > > > Option 3: Special catalog/db > > > > PROS: > > - allows shadowing built-in functions > > - allows shadowing catalog functions > > - consistent with other objects > > CONS: > > - we introduce a special namespace for built-in functions > > > > I don't see a problem with introducing the special namespace. In the end > > it is very similar to the keyword approach. In this case the catalog/db > > combination would be the "keyword" > > > > Therefore my votes: > > Option 1: -0 > > Option 2: -1 (I might change to +0 if we can come up with a better > keyword) > > Option 3: +1 > > > > Best, > > Dawid > > > > > > On Thu, 19 Sep 2019, 05:12 Xuefu Z, <usxu...@gmail.com> wrote: > > > >> Hi Aljoscha, > >> > >> Thanks for the summary and these are great questions to be answered. The > >> answer to your first question is clear: there is a general agreement to > >> override built-in functions with temp functions. > >> > >> However, your second and third questions are sort of related, as a > >> function > >> reference can be either just function name (like "func") or in the form > or > >> "cat.db.func". When a reference is just function name, it can mean > either > >> a > >> built-in function or a function defined in the current cat/db. If we > >> support overriding a built-in function with a temp function, such > >> overriding can also cover a function in the current cat/db. > >> > >> I think what Timo referred as "overriding a catalog function" means a > temp > >> function defined as "cat.db.func" overrides a catalog function "func" in > >> cat/db even if cat/db is not current. To support this, temp function has > >> to > >> be tied to a cat/db. What's why I said above that the 2nd and 3rd > >> questions > >> are related. The problem with such support is the ambiguity when user > >> defines a function w/o namespace, "CREATE TEMPORARY FUNCTION func ...". > >> Here "func" can means a global temp function, or a temp function in > >> current > >> cat/db. If we can assume the former, this creates an inconsistency > because > >> "CREATE FUNCTION func" actually means a function in current cat/db. If > we > >> assume the latter, then there is no way for user to create a global temp > >> function. > >> > >> Giving a special namespace for built-in functions may solve the > ambiguity > >> problem above, but it also introduces artificial catalog/database that > >> needs special treatment and pollutes the cleanness of the code. I would > >> rather introduce a syntax in DDL to solve the problem, like "CREATE > >> [GLOBAL] TEMPORARY FUNCTION func". > >> > >> Thus, I'd like to summarize a few candidate proposals for voting > purposes: > >> > >> 1. Support only global, temporary functions without namespace. Such temp > >> functions overrides built-in functions and catalog functions in current > >> cat/db. The resolution order is: temp functions -> built-in functions -> > >> catalog functions. (Partially or fully qualified functions has no > >> ambiguity!) > >> > >> 2. In addition to #1, support creating and referencing temporary > functions > >> associated with a cat/db with "GLOBAL" qualifier in DDL for global temp > >> functions. The resolution order is: global temp functions -> built-in > >> functions -> temp functions in current cat/db -> catalog function. > >> (Resolution for partially or fully qualified function reference is: temp > >> functions -> persistent functions.) > >> > >> 3. In addition to #1, support creating and referencing temporary > functions > >> associated with a cat/db with a special namespace for built-in functions > >> and global temp functions. The resolution is the same as #2, except that > >> the special namespace might be prefixed to a reference to a built-in > >> function or global temp function. (In absence of the special namespace, > >> the > >> resolution order is the same as in #2.) > >> > >> My personal preference is #1, given the unknown use case and introduced > >> complexity for #2 and #3. However, #2 is an acceptable alternative. > Thus, > >> my votes are: > >> > >> +1 for #1 > >> +0 for #2 > >> -1 for #3 > >> > >> Everyone, please cast your vote (in above format please!), or let me > know > >> if you have more questions or other candidates. > >> > >> Thanks, > >> Xuefu > >> > >> > >> > >> > >> > >> > >> > >> On Wed, Sep 18, 2019 at 6:42 AM Aljoscha Krettek <aljos...@apache.org> > >> wrote: > >> > >> > Hi, > >> > > >> > I think this discussion and the one for FLIP-64 are very connected. To > >> > resolve the differences, think we have to think about the basic > >> principles > >> > and find consensus there. The basic questions I see are: > >> > > >> > - Do we want to support overriding builtin functions? > >> > - Do we want to support overriding catalog functions? > >> > - And then later: should temporary functions be tied to a > >> > catalog/database? > >> > > >> > I don’t have much to say about these, except that we should somewhat > >> stick > >> > to what the industry does. But I also understand that the industry is > >> > already very divided on this. > >> > > >> > Best, > >> > Aljoscha > >> > > >> > > On 18. Sep 2019, at 11:41, Jark Wu <imj...@gmail.com> wrote: > >> > > > >> > > Hi, > >> > > > >> > > +1 to strive for reaching consensus on the remaining topics. We are > >> > close to the truth. It will waste a lot of time if we resume the topic > >> some > >> > time later. > >> > > > >> > > +1 to “1-part/override” and I’m also fine with Timo’s “cat.db.fun” > way > >> > to override a catalog function. > >> > > > >> > > I’m not sure about “system.system.fun”, it introduces a nonexistent > >> cat > >> > & db? And we still need to do special treatment for the dedicated > >> > system.system cat & db? > >> > > > >> > > Best, > >> > > Jark > >> > > > >> > > > >> > >> 在 2019年9月18日，06:54，Timo Walther <twal...@apache.org> 写道： > >> > >> > >> > >> Hi everyone, > >> > >> > >> > >> @Xuefu: I would like to avoid adding too many things incrementally. > >> > Users should be able to override all catalog objects consistently > >> according > >> > to FLIP-64 (Support for Temporary Objects in Table module). If > functions > >> > are treated completely different, we need more code and special cases. > >> From > >> > an implementation perspective, this topic only affects the lookup > logic > >> > which is rather low implementation effort which is why I would like to > >> > clarify the remaining items. As you said, we have a slight consenus on > >> > overriding built-in functions; we should also strive for reaching > >> consensus > >> > on the remaining topics. > >> > >> > >> > >> @Dawid: I like your idea as it ensures registering catalog objects > >> > consistent and the overriding of built-in functions more explicit. > >> > >> > >> > >> Thanks, > >> > >> Timo > >> > >> > >> > >> > >> > >> On 17.09.19 11:59, kai wang wrote: > >> > >>> hi, everyone > >> > >>> I think this flip is very meaningful. it supports functions that > >> can be > >> > >>> shared by different catalogs and dbs, reducing the duplication of > >> > functions. > >> > >>> > >> > >>> Our group based on flink's sql parser module implements create > >> function > >> > >>> feature, stores the parsed function metadata and schema into > mysql, > >> and > >> > >>> also customizes the catalog, customizes sql-client to support > custom > >> > >>> schemas and functions. Loaded, but the function is currently > global, > >> > and is > >> > >>> not subdivided according to catalog and db. > >> > >>> > >> > >>> In addition, I very much hope to participate in the development of > >> this > >> > >>> flip, I have been paying attention to the community, but found it > is > >> > more > >> > >>> difficult to join. > >> > >>> thank you. > >> > >>> > >> > >>> Xuefu Z <usxu...@gmail.com> 于2019年9月17日周二 上午11:19写道： > >> > >>> > >> > >>>> Thanks to Tmo and Dawid for sharing thoughts. > >> > >>>> > >> > >>>> It seems to me that there is a general consensus on having temp > >> > functions > >> > >>>> that have no namespaces and overwrite built-in functions. (As a > >> side > >> > note > >> > >>>> for comparability, the current user defined functions are all > >> > temporary and > >> > >>>> having no namespaces.) > >> > >>>> > >> > >>>> Nevertheless, I can also see the merit of having namespaced temp > >> > functions > >> > >>>> that can overwrite functions defined in a specific cat/db. > However, > >> > this > >> > >>>> idea appears orthogonal to the former and can be added > >> incrementally. > >> > >>>> > >> > >>>> How about we first implement non-namespaced temp functions now > and > >> > leave > >> > >>>> the door open for namespaced ones for later releases as the > >> > requirement > >> > >>>> might become more crystal? This also helps shorten the debate and > >> > allow us > >> > >>>> to make some progress along this direction. > >> > >>>> > >> > >>>> As to Dawid's idea of having a dedicated cat/db to host the > >> temporary > >> > temp > >> > >>>> functions that don't have namespaces, my only concern is the > >> special > >> > >>>> treatment for a cat/db, which makes code less clean, as evident > in > >> > treating > >> > >>>> the built-in catalog currently. > >> > >>>> > >> > >>>> Thanks, > >> > >>>> Xuefiu > >> > >>>> > >> > >>>> On Mon, Sep 16, 2019 at 5:07 PM Dawid Wysakowicz < > >> > >>>> wysakowicz.da...@gmail.com> > >> > >>>> wrote: > >> > >>>> > >> > >>>>> Hi, > >> > >>>>> Another idea to consider on top of Timo's suggestion. How about > we > >> > have a > >> > >>>>> special namespace (catalog + database) for built-in objects? > This > >> > catalog > >> > >>>>> would be invisible for users as Xuefu was suggesting. > >> > >>>>> > >> > >>>>> Then users could still override built-in functions, if they > fully > >> > qualify > >> > >>>>> object with the built-in namespace, but by default the common > >> logic > >> > of > >> > >>>>> current dB & cat would be used. > >> > >>>>> > >> > >>>>> CREATE TEMPORARY FUNCTION func ... > >> > >>>>> registers temporary function in current cat & dB > >> > >>>>> > >> > >>>>> CREATE TEMPORARY FUNCTION cat.db.func ... > >> > >>>>> registers temporary function in cat db > >> > >>>>> > >> > >>>>> CREATE TEMPORARY FUNCTION system.system.func ... > >> > >>>>> Overrides built-in function with temporary function > >> > >>>>> > >> > >>>>> The built-in/system namespace would not be writable for > permanent > >> > >>>> objects. > >> > >>>>> WDYT? > >> > >>>>> > >> > >>>>> This way I think we can have benefits of both solutions. > >> > >>>>> > >> > >>>>> Best, > >> > >>>>> Dawid > >> > >>>>> > >> > >>>>> > >> > >>>>> On Tue, 17 Sep 2019, 07:24 Timo Walther, <twal...@apache.org> > >> wrote: > >> > >>>>> > >> > >>>>>> Hi Bowen, > >> > >>>>>> > >> > >>>>>> I understand the potential benefit of overriding certain > built-in > >> > >>>>>> functions. I'm open to such a feature if many people agree. > >> > However, it > >> > >>>>>> would be great to still support overriding catalog functions > with > >> > >>>>>> temporary functions in order to prototype a query even though a > >> > >>>>>> catalog/database might not be available currently or should not > >> be > >> > >>>>>> modified yet. How about we support both cases? > >> > >>>>>> > >> > >>>>>> CREATE TEMPORARY FUNCTION abs > >> > >>>>>> -> creates/overrides a built-in function and never consideres > >> > current > >> > >>>>>> catalog and database; inconsistent with other DDL but > acceptable > >> for > >> > >>>>>> functions I guess. > >> > >>>>>> CREATE TEMPORARY FUNCTION cat.db.fun > >> > >>>>>> -> creates/overrides a catalog function > >> > >>>>>> > >> > >>>>>> Regarding "Flink don't have any other built-in objects (tables, > >> > views) > >> > >>>>>> except functions", this might change in the near future. Take > >> > >>>>>> https://issues.apache.org/jira/browse/FLINK-13900 as an > example. > >> > >>>>>> > >> > >>>>>> Thanks, > >> > >>>>>> Timo > >> > >>>>>> > >> > >>>>>> On 14.09.19 01:40, Bowen Li wrote: > >> > >>>>>>> Hi Fabian, > >> > >>>>>>> > >> > >>>>>>> Yes, I agree 1-part/no-override is the least favorable thus I > >> > didn't > >> > >>>>>>> include that as a voting option, and the discussion is mainly > >> > between > >> > >>>>>>> 1-part/override builtin and 3-part/not override builtin. > >> > >>>>>>> > >> > >>>>>>> Re > However, it means that temp functions are differently > >> treated > >> > >>>> than > >> > >>>>>>> other db objects. > >> > >>>>>>> IMO, the treatment difference results from the fact that > >> functions > >> > >>>> are > >> > >>>>> a > >> > >>>>>>> bit different from other objects - Flink don't have any other > >> > >>>> built-in > >> > >>>>>>> objects (tables, views) except functions. > >> > >>>>>>> > >> > >>>>>>> Cheers, > >> > >>>>>>> Bowen > >> > >>>>>>> > >> > >>>>>> > >> > >>>> > >> > >>>> -- > >> > >>>> Xuefu Zhang > >> > >>>> > >> > >>>> "In Honey We Trust!" > >> > >>>> > >> > >> > >> > > > >> > > >> > > >> > >> -- > >> Xuefu Zhang > >> > >> "In Honey We Trust!" > >> > > > -- Xuefu Zhang "In Honey We Trust!"
