[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279423#comment-17279423
]
Jane Chan edited comment on FLINK-21045 at 2/5/21, 7:28 AM:
Hi [~nicholasjiang], "Extend Core Table System with Pluggable Modules" is the
title of FLIP-68 and FLINK-14132.
How about
{panel:title=Improve Usability of Pluggable Modules}
*Description*
This improvement aims to
# Simplify the module discovery mapping by module name. This will encourage
users to create singleton of module instances.
# Support changing the resolution order of modules in a flexible manner. This
will introduce two methods {{#useModules}} and {{#listFullModules}} in both
{{ModuleManager}} and {{TableEnvironment}}.
# Support SQL syntax upon {{LOAD/UNLOAD MODULE}}, {{USE MODULES}}, and {{SHOW
[FULL] MODULES}} in both {{FlinkSqlParserImpl}} and {{SqlClient}}.
# Update the documentation to keep users informed of this improvement.
Please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html]
for more details.
{panel}
The proposed subtasks follow the description bullet list. What do you think?
was (Author: qingyue):
Hi [~nicholasjiang], "Extend Core Table System with Pluggable Modules" is the
title of FLIP-68 and FLINK-14132.
How about
{panel:title=Improve Usability of Pluggable Modules}
*Description*
This improvement aims to
# Simplify the module discovery mapping by module name. This will encourage
users to create singleton of module instance.
# Support changing the resolution order of modules in a flexible manner. This
will introduce two methods {{#useModules}} and {{#listFullModules}} in both
{{ModuleManager}} and {{TableEnvironment}}.
# Support SQL syntax upon {{LOAD/UNLOAD MODULE}}, {{USE MODULES}}, and {{SHOW
[FULL] MODULES in both {{SqlParserImpl}} and {{SqlClient}}.
# Update the documentation to keep users informed with this improvement.
Please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html]
for more details.
{panel}
The proposed subtasks follows the description bullet list. What do you think?
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Nicholas Jiang
>Assignee: Jane Chan
>Priority: Major
> Fix For: 1.13.0
>
>
> At present, Flink SQL doesn't support the 'load module' and 'unload module'
> SQL syntax. It's necessary for uses in the situation that users load and
> unload user-defined module through table api or sql client.
> SQL syntax has been proposed in FLIP-68:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 12:43 PM:
-
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}.
Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and
{{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean used;
ModuleEntry(String name, boolean used) {
this.name = name;
this.used = used;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods {{listFullModules()}} and {{useModules(String…
names)}} to {{TableEnvironment}} interface. The impl will simply call
{{moduleManager#listFullModules()}} and {{moduleManager#useModules(String…
names)}}.
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate {{ModuleFactory}} to extends
{{org.apache.flink.table.factories.Factory}}. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name [WITH ('prop' = 'myProp', ...)]{code}
Explanation
* Module name {{module_name}} is a simple identifier instead of a string
literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
---
Flink SQL> LOAD MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
| ml | false |
---
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
---
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | false |
---
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW F
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 12:10 PM:
-
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}.
Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and
{{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean used;
ModuleEntry(String name, boolean used) {
this.name = name;
this.used = used;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods {{listFullModules()}} and {{useModules(String…
names)}} to {{TableEnvironment}} interface. The impl will simply call
{{moduleManager#listFullModules()}} and {{moduleManager#useModules(String…
names)}}.
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate {{ModuleFactory}} to extends
{{org.apache.flink.table.factories.Factory}}. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name [WITH ('prop' = 'myProp', ...)]{code}
Explanation
* Module name {{module_name}} is a simple identifier instead of a string
literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
---
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
| ml | false |
---
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
---
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | false |
---
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 11:46 AM:
-
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}.
Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and
{{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean status;
ModuleEntry(String name, boolean status) {
this.name = name;
this.status = status;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods listFullModules() and useModules(String… names) to
TableEnvironment interface. The impl will simply call
moduleManager#listFullModules() and moduleManager#useModules(String… names).
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name WITH ('prop' = 'myProp', ...){code}
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
---
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
| ml | false |
---
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
---
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | false |
---
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MO
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 10:41 AM:
-
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
h2. API Change
h3. ModuleManager
We introduce two methods to support USE MODULES and SHOW FULL MODULES. Beyond
that, the behavior of methods listModules, listFunctions, and
getFunctionDefinition will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(List names) {
// list all loaded modules with status
}
}
{code}
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
|LOAD MODULE module_name WITH ('prop' = 'myProp', ...)|
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
---
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
|UNLOAD MODULE module_name|
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
| ml | false |
---
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
---
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
|USE MODULES x [,y, z, ...]|
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | false |
---
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| hive | true |
| ml | true |
| core | false |
---
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| hive | true |
| ml | true |
| core | false |
---
{code}
h3. SHOW [FULL] MODULES
Display the loa
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 10:18 AM:
-
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
h2. API Change
h3. ModuleManager
We introduce two methods to support USE MODULES and SHOW FULL MODULES. Beyond
that, the behavior of methods listModules, listFunctions, and
getFunctionDefinition will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(List names) {
// list all loaded modules with status
}
}
{code}
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
|LOAD MODULE module_name WITH ('prop' = 'myProp', ...)|
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
---
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
|UNLOAD MODULE module_name|
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | true |
---
Flink SQL> UNLOAD MODULE ml;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
|USE MODULES x [,y, z, ...]|
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | false |
---
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| hive | true |
| ml | true |
| core | false |
---
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| hive | true |
| ml | true |
| core | false |
---
{code}
h3. SHOW [FULL] MODULES
Display the loaded mod
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 10:10 AM:
-
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
h2. API Change
h3. ModuleManager
We introduce two methods to support USE MODULES and SHOW FULL MODULES. Beyond
that, the behavior of methods listModules, listFunctions, and
getFunctionDefinition will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(List names) {
// list all loaded modules with status
}
}
{code}
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
|LOAD MODULE module_name WITH ('prop' = 'myProp', ...)|
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
---
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
|UNLOAD MODULE module_name|
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | true |
---
Flink SQL> UNLOAD MODULE ml;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| hive | true |
---
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
|USE MODULES x [,y, z, ...]|
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| core | true |
| ml | false |
| hive | false |
---
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| hive | true |
| ml | true |
| core | false |
---
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
---
| modules | used |
---
| hive | true |
| ml | true |
| core | false |
---
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
|SHOW [FULL] MODULES|
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FUL
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271508#comment-17271508
]
Jane Chan edited comment on FLINK-21045 at 1/29/21, 7:44 PM:
-
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
According to FLIP-68
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and this improvement
will not involve dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because <1> if the module is
loaded before, an exception will be thrown. <2> if it's a new module, then
without specifying properties the TableFactoryService cannot find the suitable
factory. Thus the optional `{{WITH}}` may be necessary instead? On the other
hand, if users want to shift the resolution order on loaded modules, it could
be concise to only provide the module names rather than a long property list,
how about providing `{{RELOAD MODULE X}}` instead of `{{LOAD MODULE X}}`, which
describes changing the resolution order of a loaded module {{X}}. (It may
beyond the scope of this improvement, I'm just wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
was (Author: qingyue):
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
According to FLIP-68
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and this improvement
will not involve dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because <1> if the module is
loaded before, an exception will be thrown. <2> if it's a new module, then
without specifying properties the TableFactoryService cannot find the suitable
factory. Thus the optional `{{WITH}}` may be necessary instead? (Or throw an
exception is ok?) On the other hand, if we want to describe an operation
performed on module name only, is it possible to`{{RELOAD MODULE X}}` instead
of `{{LOAD MODULE X}}`, which describes changing the resolution order of a
loaded module {{X}}. (It may beyond the scope of this improvement, I'm just
wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table S
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271508#comment-17271508
]
Jane Chan edited comment on FLINK-21045 at 1/29/21, 7:41 PM:
-
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
According to FLIP-68
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and this improvement
will not involve dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because <1> if the module is
loaded before, an exception will be thrown. <2> if it's a new module, then
without specifying properties the TableFactoryService cannot find the suitable
factory. Thus the optional `{{WITH}}` may be necessary instead? (Or throw an
exception is ok?) On the other hand, if we want to describe an operation
performed on module name only, is it possible to`{{RELOAD MODULE X}}` instead
of `{{LOAD MODULE X}}`, which describes changing the resolution order of a
loaded module {{X}}. (It may beyond the scope of this improvement, I'm just
wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
was (Author: qingyue):
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
Although
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and there's nothing
to do with dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because the only feasible
operation for `{{LOAD}}` is to load a new module. Thus the optional `{{WITH}}`
may be necessary instead? (Or throw an exception is ok?) On the other hand, if
we want to describe an operation performed on module name only, is it possible
to`{{RELOAD MODULE X}}` instead of `{{LOAD MODULE X}}`, which describes
changing the resolution order of a loaded module {{X}}. (It may beyond the
scope of this improvement, I'm just wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Nicholas Jiang
>Assignee: Jane Chan
>Priority: Major
> Fix For: 1.13.0
>
>
> At present, Fli
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[ https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271910#comment-17271910 ] Nicholas Jiang edited comment on FLINK-21045 at 1/26/21, 7:13 AM: -- [~qingyue], [~jark], IMO, the WITH is more suitable to `CREATE MODULE` syntax, doesn't make much sense for `LOAD MODULE`. `LOAD MODULE` syntax only needs to specify the module name. was (Author: nicholasjiang): [~qingyue][~jark], IMO, the WITH is more suitable to `CREATE MODULE` syntax, doesn't make much sense for `LOAD MODULE`. `LOAD MODULE` syntax only needs to specify the module name. > Support 'load module' and 'unload module' SQL syntax > > > Key: FLINK-21045 > URL: https://issues.apache.org/jira/browse/FLINK-21045 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Nicholas Jiang >Assignee: Jane Chan >Priority: Major > Fix For: 1.13.0 > > > At present, Flink SQL doesn't support the 'load module' and 'unload module' > SQL syntax. It's necessary for uses in the situation that users load and > unload user-defined module through table api or sql client. > SQL syntax has been proposed in FLIP-68: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271508#comment-17271508
]
Jane Chan edited comment on FLINK-21045 at 1/26/21, 1:47 AM:
-
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
Although
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and there's nothing
to do with dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because the only feasible
operation for `{{LOAD}}` is to load a new module. Thus the optional `{{WITH}}`
may be necessary instead? (Or throw an exception is ok?) On the other hand, if
we want to describe an operation performed on module name only, is it possible
to`{{RELOAD MODULE X}}` instead of `{{LOAD MODULE X}}`, which describes
changing the resolution order of a loaded module {{X}}. (It may beyond the
scope of this improvement, I'm just wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
was (Author: qingyue):
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
Although
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and there's nothing
to do with dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because the only feasible
operation for `{{LOAD}}` is to load a new module. Thus the optional `{{WITH}}`
may be necessary instead? On the other hand, if we want to describe an
operation performed on module name only, is it possible to`{{RELOAD MODULE X}}`
instead of `{{LOAD MODULE X}}`, which describes changing the resolution order
of a loaded module {{X}}. (It may beyond the scope of this improvement, I'm
just wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Nicholas Jiang
>Assignee: Jane Chan
>Priority: Major
> Fix For: 1.13.0
>
>
> At present, Flink SQL doesn't support the 'load module' and 'unload module'
> SQL syntax. It's necessary for uses in the situation that users load and
> unload user-defined module th
