[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Reviewed-on: http://gerrit.cloudera.org:8080/19252 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 172 insertions(+), 4 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 25 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 24: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 24 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 06 Dec 2022 06:56:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 24: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11969/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 24 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 06 Dec 2022 02:03:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 24: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8867/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 24 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 06 Dec 2022 01:45:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 24: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/19252/23/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/19252/23/be/src/service/query-options.cc@1041 PS23, Line 1041: ); > Don't need c_str() here. __set_fallback_db_for_functions() accept a "const Done -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 24 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 06 Dec 2022 01:44:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Quanlong Huang has uploaded a new patch set (#24) to the change originally created by Xiaoqing Gao. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 172 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/24 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 24 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 23: (1 comment) > Patch Set 22: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8853/ I think the failure is unrelated. Filed IMPALA-11775 to track it. http://gerrit.cloudera.org:8080/#/c/19252/23/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/19252/23/be/src/service/query-options.cc@1041 PS23, Line 1041: .c_str() Don't need c_str() here. __set_fallback_db_for_functions() accept a "const std::string&". We use 'value' directly in other string options, e.g. https://github.com/apache/impala/blob/63a344066997c5e4695cbdffeb5bf9a17e532d4b/be/src/service/query-options.cc#L259 https://github.com/apache/impala/blob/63a344066997c5e4695cbdffeb5bf9a17e532d4b/be/src/service/query-options.cc#L304 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 23 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Mon, 05 Dec 2022 01:34:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 22: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8853/ -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 22 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 13:56:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 22: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 22 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 08:52:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 22: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8853/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 22 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 08:52:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 21: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 21 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 08:51:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11945/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 21 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 07:27:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 20: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11944/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 20 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 07:24:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Xiaoqing Gao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 15: (2 comments) http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java File fe/src/main/java/org/apache/impala/analysis/FunctionName.java: http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@128 PS15, Line 128: preferBuiltinsDb > Thanks for the confirmation! Thanks for the suggestion, I modified the annotation on the function. http://gerrit.cloudera.org:8080/#/c/19252/19/fe/src/main/java/org/apache/impala/analysis/FunctionName.java File fe/src/main/java/org/apache/impala/analysis/FunctionName.java: http://gerrit.cloudera.org:8080/#/c/19252/19/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@146 PS19, Line 146: return feDb != null && feDb.containsFunction(fn_); > Could we add a code comment here pointing out that to execute a UDF of the Done -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 07:13:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#21). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 172 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/21 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 21 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#20). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 171 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/20 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 20 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 20: (1 comment) http://gerrit.cloudera.org:8080/#/c/19252/20/fe/src/main/java/org/apache/impala/analysis/FunctionName.java File fe/src/main/java/org/apache/impala/analysis/FunctionName.java: http://gerrit.cloudera.org:8080/#/c/19252/20/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@105 PS20, Line 105:* - Else if the query option of 'FALLBACK_DB_FOR_FUNCTIONS' being set and the function line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 20 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 07:05:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 19: Code-Review+1 (2 comments) Thanks for the effort Xiaoqing and Csaba! I only have 2 minor comments and I do not have any other suggestion. http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java File fe/src/main/java/org/apache/impala/analysis/FunctionName.java: http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@128 PS15, Line 128: ltinsDb.NAME; > Yes, the query option doesn't take effect to CREATE/DROP FUNCTION statement Thanks for the confirmation! We probably don't have to consider the CREATE/DROP FUNCTION statements since this use case may not be that common. In that case, could we add 2 more code comments as described in the following if they are reasonable? 1. A code comment for the else-if branch pointing out that this branch is for the SELECT statement with the query option of 'FALLBACK_DB_FOR_FUNCTIONS' being set because for the SELECT statement, 'preferBuiltinsDb' is true. 2. A code comment for the else branch pointing out that this branch is for a) the SELECT statement without the query option of 'FALLBACK_DB_FOR_FUNCTIONS' being set, or b) the CREATE/DROP FUNCTION statements whether or not 'FALLBACK_DB_FOR_FUNCTIONS' is specified. http://gerrit.cloudera.org:8080/#/c/19252/19/fe/src/main/java/org/apache/impala/analysis/FunctionName.java File fe/src/main/java/org/apache/impala/analysis/FunctionName.java: http://gerrit.cloudera.org:8080/#/c/19252/19/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@146 PS19, Line 146: FeDb feDb = analyzer.getDb(dbName, Privilege.VIEW_METADATA, false); Could we add a code comment here pointing out that to execute a UDF of the fallback database in a SELECT statement, the requesting user has be to granted any one of the INSERT, REFRESH, SELECT privileges on the fallback database? -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 02 Dec 2022 04:25:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has removed a vote on this change. Change subject: IMPALA-11728: Set fallback database for functions .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 19: Code-Review+2 The test failure looks like an unrelated flaky test. Upgrading to +2 will run the GVO tomorrow of no one has comments. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 18:26:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 19: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8848/ -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 16:55:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11933/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 12:10:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 19: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8848/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 11:52:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#19). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 165 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/19 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 19 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
lipeng...@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 18: Code-Review+1 Look good to me! Thanks for fixing this. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 18 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 10:03:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 18: Code-Review+1 (1 comment) Thanks a lot for fixing the authorization issue! lgtm, can upgrade to +2 and start the tests if others have no comments http://gerrit.cloudera.org:8080/#/c/19252/18/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/19252/18/tests/authorization/test_ranger.py@1146 PS18, Line 1146: use typo: user -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 18 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 09:47:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Xiaoqing Gao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 15: (2 comments) http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java File fe/src/main/java/org/apache/impala/analysis/FunctionName.java: http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@128 PS15, Line 128: preferBuiltinsDb > Is it possible that 'preferBuiltinsDb' is false but fallbackDbContainsFn(an Yes, the query option doesn't take effect to CREATE/DROP FUNCTION statements. The source of this requirement is that the user-defined function only needs to be created once, and it takes effect on all databases. After setting FALLBACK_DB_FOR_FUNCTIONS, the queries use functions without specified database. For CREATE/DROP FUNCTION statements. I did not find the required scene. What do you think? http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@145 PS15, Line 145: FeDb feDb = analyzer.getDb(dbName, false); > To address Csaba's concern at https://gerrit.cloudera.org/c/19252/15/tests/ Thanks for your very useful advice. I changed it and added the test. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 08:37:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11931/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 18 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 01 Dec 2022 08:17:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#18). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 165 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/18 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 18 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Fang-Yu Rao has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: IMPALA-11728: Set fallback database for functions
..
Patch Set 17:
(3 comments)
Hi all, I just left some more comments since the JIRA seems to be more
complicated than I expected. Sorry that I did not realize it in my previous
review. Let me know if my feedback is reasonable. Thanks!
http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java
File fe/src/main/java/org/apache/impala/analysis/FunctionName.java:
http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@128
PS15, Line 128: preferBuiltinsDb
Is it possible that 'preferBuiltinsDb' is false but
fallbackDbContainsFn(analyzer) is true?
I just found that in the CREATE/DROP FUNCTION statements, 'preferBuiltinsDb'
could be false. Do we need to consider those 2 statements as well in this JIRA?
It seems that for the CREATE/DROP FUNCTION statements, the current patch always
uses the database returned by "analyzer.getDefaultDb()". Is my understanding
correct?
http://gerrit.cloudera.org:8080/#/c/19252/15/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@145
PS15, Line 145: FeDb feDb = analyzer.getDb(dbName, false);
To address Csaba's concern at
https://gerrit.cloudera.org/c/19252/15/tests/authorization/test_ranger.py#1151,
we could probably change this to the following if we only want to consider the
case when a user tries to execute UDF's in a SELECT query. This way the user
has to be granted any of the INSERT, REFRESH, SELECT privileges on the fallback
database even though a) the database does not exist or b) the database exists
but the function does not.
FeDb feDb = analyzer.getDb(dbName, Privilege.VIEW_METADATA, /*
throwIfDoesNotExist */ false);
However, if we would like to additionally consider the CREATE/DROP FUNCTION
statements, then different privileges have to be registered.
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/authorization/test_ranger.py@1146
PS15, Line 1146: result = self.execute_query_expect_failure(
: non_owner_client, "select identity(1)",
query_options={
: 'FALLBACK_DB_FOR_FUNCTIONS': unique_database},
user=test_user)
: err = "User '{0}' does not have privileges to access:
{1}".format(
: test_user, unique_database)
: assert err in str(result)
> It makes sense. But currently it throw an Exception "default.fn() unknown f
Thanks Xiaoqing for adding the test case and Csaba for pointing out a subtle
case when a requesting user who explicitly specifies a fallback database
'FALLBACK_DB_FOR_FUNCTIONS' that does not contain the function! I did not
discover this in my previous review.
I think it should be possible to not change the current precedence proposed by
Xiaoqing. Specifically, in FunctionName#FALLBACK_DB_FOR_FUNCTIONS(), instead of
calling "analyzer.getDb(dbName, /* throwIfDoesNotExist */ false)", we call
"analyzer.getDb(dbName, Privilege.VIEW_METADATA, /* throwIfDoesNotExist */
false)". This way we also register a privilege request for the fallback
database even though a) the database does not exist or b) the database exists
but the function does not. As a result, Impala's frontend will still check with
the Ranger plugin to see if the requesting user is granted any of the REFRESH,
INSERT, SELECT privileges on the fallback database. This works if
FunctionName#fallbackDbContainsFn() is only invoked in a SELECT query in that
to execute a UDF of a database in a SELECT query, Impala currently requires the
requesting user to be granted any of those 3 privileges on the database as
mentioned above.
However, if we need to consider the CREATE/DROP FUNCTION statements as well in
this JIRA, we will have to register different privilege requests accordingly.
Hence, depending on whether we would like to consider the CREATE/DROP FUNCTION
statements in this JIRA, different test cases should be added.
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 17
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Wed, 30 Nov 2022 10:09:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11923/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 17 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 30 Nov 2022 07:11:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11922/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 16 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 30 Nov 2022 07:04:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#17). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 162 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/17 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 17 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/19252/16/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/19252/16/tests/query_test/test_udfs.py@682 PS16, Line 682: = flake8: E711 comparison to None should be 'if cond is None:' -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 16 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 30 Nov 2022 06:46:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Xiaoqing Gao has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: IMPALA-11728: Set fallback database for functions
..
Patch Set 15:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/authorization/test_ranger.py@1146
PS15, Line 1146: result = self.execute_query_expect_failure(
: non_owner_client, "select identity(1)",
query_options={
: 'FALLBACK_DB_FOR_FUNCTIONS': unique_database},
user=test_user)
: err = "User '{0}' does not have privileges to access:
{1}".format(
: test_user, unique_database)
: assert err in str(result)
> Can you check what error is returned if look for a function that does not e
It makes sense. But currently it throw an Exception "default.fn() unknown for
database default.". Because it use the default database if the function does
not exit in fallback db.
It will throw an Exception "User non_owner does not have privileges to access:
test_util_db" if change the exact precedence of the functions.
1. _impala_builtins
2. whether the function exists in fallback db, use fallback db after setting
FALLBACK_DB_FOR_FUNCTIONS, else use function in current db if not set
FALLBACK_DB_FOR_FUNCTIONS.
If modify the precedence, there is a disadvantage that users cannot access
their own databases if set FALLBACK_DB_FOR_FUNCTIONS.
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/query_test/test_udfs.py
File tests/query_test/test_udfs.py:
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/query_test/test_udfs.py@674
PS15, Line 674:
> Can you also add a test for show function? It should return empty result, e
Done
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 15
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Wed, 30 Nov 2022 06:46:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#16). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 162 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/16 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 16 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 29 Nov 2022 16:58:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: IMPALA-11728: Set fallback database for functions
..
Patch Set 15:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/authorization/test_ranger.py@1146
PS15, Line 1146: result = self.execute_query_expect_failure(
: non_owner_client, "select identity(1)",
query_options={
: 'FALLBACK_DB_FOR_FUNCTIONS': unique_database},
user=test_user)
: err = "User '{0}' does not have privileges to access:
{1}".format(
: test_user, unique_database)
: assert err in str(result)
Can you check what error is returned if look for a function that does not exist
in fallback db? Generally we should not be able to decide based on the error
message whether something (e.g. table, function) exists or not if the user has
no privileges on it at all (this why an auth error is returned in line 1132.
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/query_test/test_udfs.py
File tests/query_test/test_udfs.py:
http://gerrit.cloudera.org:8080/#/c/19252/15/tests/query_test/test_udfs.py@674
PS15, Line 674:
Can you also add a test for show function? It should return empty result, even
when FALLBACK_DB_FOR_FUNCTIONS is set and is not empty.
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 15
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Tue, 29 Nov 2022 14:21:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8841/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 29 Nov 2022 11:49:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Xiaoqing Gao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 15: Thanks for your valuable suggestion! I added FE and e2e tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 29 Nov 2022 09:28:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11919/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 29 Nov 2022 09:21:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11918/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 14 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 29 Nov 2022 09:19:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#15). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 149 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/15 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 15 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, Fang-Yu Rao, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#14). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Added various FE and E2E tests to verify Impala's behavior after setting FALLBACK_DB_FOR_FUNCTIONS. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M tests/authorization/test_ranger.py M tests/query_test/test_udfs.py 9 files changed, 149 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/14 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 14 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 14: (4 comments) http://gerrit.cloudera.org:8080/#/c/19252/14/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/19252/14/tests/authorization/test_ranger.py@1141 PS14, Line 1141: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/19252/14/tests/authorization/test_ranger.py@1141 PS14, Line 1141: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/14/tests/authorization/test_ranger.py@1158 PS14, Line 1158: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/14/tests/authorization/test_ranger.py@1158 PS14, Line 1158: # A user is allowed to use functions in the fallback database if the user is line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 14 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 29 Nov 2022 09:00:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: I left a minor comment regarding how to add a new end-to-end test against Impala's behavior with respect to authorization when the newly introduced query option is set. I do not have any other suggestion. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 25 Nov 2022 06:38:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: > Patch Set 13: -Code-Review > > > Just a minor question, FALLBACK_DB_FOR_FUNCTIONS brings flexibility > > to FUNCTION calls, but it breaks the isolation between different > > databases, we can still use the Apache Ranger for authorization of > > those FUNCTIONs to implement the isolation, right? Has this been > > tested? > > That is a very good point, Impala should not allow using functions in the > fallback database unless the user has some privileges on the given database. > > Note that there is an open review related to the authorization of UDFs: > https://gerrit.cloudera.org/#/c/19194/ > > An example authorization test which deals with UDF + sets a query option: > https://github.com/apache/impala/blob/52956bae141acf2ecdd7b28ff7edb4d2f2fe3f10/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java#L2932 > > Note that the AuthorizationStmtTest test suite can be extremely slow to run. We could probably add a new end-to-end test in tests/authorization/test_ranger.py as briefly described in the following if we would like to verify Impala's behavior with respect to authorization when the query option of 'FALLBACK_DB_FOR_FUNCTIONS' is set. 1. As the user 'admin', execute "create database test_db". 2. As the user 'admin', execute "create function test_db.identity(bigint) returns bigint location '/test-warehouse/impala-hive-udfs.jar' symbol='org.apache.impala.TestUdf'". 3. As the user 'non_owner', execute "select identity(1)". Verify that the error message is "ERROR: AuthorizationException: User 'non_owner' does not have privileges to access: default". 4. As the user 'admin', execute "grant refresh on database default to user non_owner" and "refresh authorization". 5. As the user 'non_owner', execute "select identity(1)". Verify that the error message is "ERROR: AnalysisException: default.identity() unknown for database default. Currently this db has 0 functions." 6. As the user 'non_owner', execute "set FALLBACK_DB_FOR_FUNCTIONS=test_db" and then "select identity(1)". Verify that the error message is "ERROR: AuthorizationException: User 'non_owner' does not have privileges to access: test_db". 7. As the user 'admin', execute "grant refresh on database test_db to user non_owner" and "refresh authorization". 8. As the user 'non_owner', execute "select identity(1)". Verify that the query could be correctly executed. That is, the result should be 1. Note that currently in order for the user to execute a UDF in the database , the requesting user has to be granted one of the SELECT, REFRESH, INSERT privileges on all the tables and all the columns in the database. This could be done by an administrator (e.g., the user 'admin') submitting to Impala server the command "grant on database to user " where is SELECT, REFRESH, or INSERT. There is a similar end-to-end test at https://gerrit.cloudera.org/c/19194/8/tests/authorization/test_ranger.py#1154 written for IMPALA-10986. The name of the new test suggested above could be something like "test_select_function_with_fallback_db". -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Fri, 25 Nov 2022 06:36:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: -Code-Review > Just a minor question, FALLBACK_DB_FOR_FUNCTIONS brings flexibility > to FUNCTION calls, but it breaks the isolation between different > databases, we can still use the Apache Ranger for authorization of > those FUNCTIONs to implement the isolation, right? Has this been > tested? That is a very good point, Impala should not allow using functions in the fallback database unless the user has some privileges on the given database. Note that there is an open review related to the authorization of UDFs: https://gerrit.cloudera.org/#/c/19194/ An example authorization test which deals with UDF + sets a query option: https://github.com/apache/impala/blob/52956bae141acf2ecdd7b28ff7edb4d2f2fe3f10/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java#L2932 Note that the AuthorizationStmtTest test suite can be extremely slow to run. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 24 Nov 2022 14:37:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
lipeng...@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: Code-Review+1 Just a minor question, FALLBACK_DB_FOR_FUNCTIONS brings flexibility to FUNCTION calls, but it breaks the isolation between different databases, we can still use the Apache Ranger for authorization of those FUNCTIONs to implement the isolation, right? Has this been tested? -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 24 Nov 2022 05:54:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 15:54:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: Code-Review+1 Thanks, this looks ok from my side. If no one else has comments for a few days, I will upgrade to +2. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 14:04:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#13). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Add test_set_fallback_db_for_functions in test_udfs.py. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/13 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8828/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 13 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 10:52:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Xiaoqing Gao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@16 PS7, Line 16: Testing: > Sorry, I didn't mean it this way, but simply adding sql queries to the comm I'm sorry I misunderstood your meaning. I added the example in commit message. http://gerrit.cloudera.org:8080/#/c/19252/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19252/11//COMMIT_MSG@9 PS11, Line 9: The cre > nit: currently Done -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 7 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 10:42:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#12). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Currently the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db An example of how to use this feature is listed below. Assuming we connected to the default database and then execute the statements. The query executed successfully after setting FALLBACK_DB_FOR_FUNCTIONS. CREATE FUNCTION util_db.fn() returns int LOCATION '/libTestUdfs.so' symbol='NoArgs'; set FALLBACK_DB_FOR_FUNCTIONS=util_db; SELECT fn() from functional.alltypes limit 1; Testing: - Add test_set_fallback_db_for_functions in test_udfs.py. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/12 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 12 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 11: Code-Review+1 (2 comments) Still some comments on the commit message, otherwise looks great. http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@16 PS7, Line 16: qualified. > I added four test case descriptions. Sorry, I didn't mean it this way, but simply adding sql queries to the commit message to show this feature can be used , so that someone who wants to use will not need to read the tests. http://gerrit.cloudera.org:8080/#/c/19252/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19252/11//COMMIT_MSG@9 PS11, Line 9: Current nit: currently -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 11 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 09:02:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#11). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Current the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db Testing: - add test_set_fallback_db_for_functions in test_udfs.py. Creating function fn() in unique_database. case 1: When the function name is fully qualified then this query option has no effect. case 2: Calling the function in default database and throwing an exception "default.fn() unknown for database default". case 3: After setting the query option FALLBACK_DB_FOR_FUNCTIONS=unique_database, the query is successful. case 4: Creating function abs(int) that also exists as builtin function, calling the function and it uses the builtin function. Because the builtin functions have higher priority. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/11 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 11 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11895/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 10 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 06:14:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11894/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 9 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 06:10:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11893/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 8 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 06:06:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#10). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Current the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db Testing: - add test_set_fallback_db_for_functions in test_udfs.py. Creating function fn() in unique_database. case 1: When the function name is fully qualified then this query option has no effect. case 2: Calling the function in default database and throwing an exception "default.fn() unknown for database default". case 3: After setting the query option FALLBACK_DB_FOR_FUNCTIONS=unique_database, the query is successful. case 4: Creating function abs(int) that also exists as builtin function, calling the function and it uses the builtin function. Because the builtin functions have higher priority. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/10 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 10 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 9: (3 comments) http://gerrit.cloudera.org:8080/#/c/19252/9/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/19252/9/tests/query_test/test_udfs.py@656 PS9, Line 656: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/19252/9/tests/query_test/test_udfs.py@671 PS9, Line 671: flake8: E999 IndentationError: unexpected indent http://gerrit.cloudera.org:8080/#/c/19252/9/tests/query_test/test_udfs.py@671 PS9, Line 671: " flake8: E113 unexpected indentation -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 9 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 05:51:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Xiaoqing Gao has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: IMPALA-11728: Set fallback database for functions
..
Patch Set 8:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@7
PS7, Line 7: : Set
> nit: missing :
Done
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@10
PS7, Line 10: When using the function in another database, it will throw an
AnalysisException
> Could add that the function is used without specifying the database. When t
I added in case 1.
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@11
PS7, Line 11: own in the current da
> Do you mean "the function is unknown in the current database"?
Yes, I changed the description.
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@13
PS7, Line 13: Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback
database
> Can you add the exact name of the query option here or in the title?
Done
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@15
PS7, Line 15:
> Could add the exact precedence of the functions:
Done
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@16
PS7, Line 16: The exact precedence of the functions:
> Some example in the commit message could make this feature very clear, e.g.
I added four test case descriptions.
http://gerrit.cloudera.org:8080/#/c/19252/7/tests/query_test/test_udfs.py
File tests/query_test/test_udfs.py:
http://gerrit.cloudera.org:8080/#/c/19252/7/tests/query_test/test_udfs.py@649
PS7, Line 649: create_function_stmt = "create function `{0}`.fn() returns
int "\
> This is not needed, as unique_database always starts empty.
Done
http://gerrit.cloudera.org:8080/#/c/19252/7/tests/query_test/test_udfs.py@677
PS7, Line 677:
> No need to cleanup, a DROP DATABASE CASCADE will be ran after the test.
Done
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 8
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Wed, 23 Nov 2022 05:50:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#9). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Current the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db Testing: - add test_set_fallback_db_for_functions in test_udfs.py. Creating function fn() in unique_database. case 1: When the function name is fully qualified then this query option has no effect. case 2: Calling the function in default database and throwing an exception "default.fn() unknown for database default". case 3: After setting the query option FALLBACK_DB_FOR_FUNCTIONS=unique_database, the query is successful. case 4: Creating function abs(int) that also exists as builtin function, calling the function and it uses the builtin function. Because the builtin functions have higher priority. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/9 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 9 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728: Set fallback database for functions .. Patch Set 8: (5 comments) http://gerrit.cloudera.org:8080/#/c/19252/8/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/19252/8/tests/query_test/test_udfs.py@654 PS8, Line 654: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/8/tests/query_test/test_udfs.py@654 PS8, Line 654: # case 1: When the function name is fully qualified then this query option line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/8/tests/query_test/test_udfs.py@656 PS8, Line 656: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/19252/8/tests/query_test/test_udfs.py@674 PS8, Line 674: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/19252/8/tests/query_test/test_udfs.py@674 PS8, Line 674: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 8 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 23 Nov 2022 05:47:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728: Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#8). Change subject: IMPALA-11728: Set fallback database for functions .. IMPALA-11728: Set fallback database for functions Current the create function statement only works on specified database. When using the function in another database, it will throw an AnalysisException because the function is unknown in the current database. Add query option FALLBACK_DB_FOR_FUNCTIONS. When setting fallback database query option, function names do not need to be fully qualified. The exact precedence of the functions: 1. _impala_builtins 2. fallback db specified in query option FALLBACK_DB_FOR_FUNCTIONS 3. function in current db Testing: - add test_set_fallback_db_for_functions in test_udfs.py. Creating function fn() in unique_database. case 1: When the function name is fully qualified then this query option has no effect. case 2: Calling the function in default database and throwing an exception "default.fn() unknown for database default". case 3: After setting the query option FALLBACK_DB_FOR_FUNCTIONS=unique_database, the query is successful. case 4: Creating function abs(int) that also exists as builtin function, calling the function and it uses the builtin function. Because the builtin functions have higher priority. Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 65 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/8 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 8 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: IMPALA-11728 Set fallback database for functions
..
Patch Set 7:
(8 comments)
The code looks good to me, some comments to make commit message clearer.
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@7
PS7, Line 7: Set f
nit: missing :
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@10
PS7, Line 10: When using the function in another database, it will throw
Could add that the function is used without specifying the database. When the
function name is fully qualified then this query option has no effect.
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@11
PS7, Line 11: unknown for database
Do you mean "the function is unknown in the current database"?
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@13
PS7, Line 13: Add a fallback database for functions as query option. It works on
Can you add the exact name of the query option here or in the title?
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@15
PS7, Line 15:
Could add the exact precedence of the functions:
1. builtins
2. fallback db
3. function in current db
http://gerrit.cloudera.org:8080/#/c/19252/7//COMMIT_MSG@16
PS7, Line 16: Testing:
Some example in the commit message could make this feature very clear, e.g.
creating a function, setting the query option, then calling the function.
http://gerrit.cloudera.org:8080/#/c/19252/7/tests/query_test/test_udfs.py
File tests/query_test/test_udfs.py:
http://gerrit.cloudera.org:8080/#/c/19252/7/tests/query_test/test_udfs.py@649
PS7, Line 649: drop_function_stmt = "drop function if exists
`{0}`.fn()".format(unique_database)
This is not needed, as unique_database always starts empty.
http://gerrit.cloudera.org:8080/#/c/19252/7/tests/query_test/test_udfs.py@677
PS7, Line 677: self.client.execute(drop_function_stmt)
No need to cleanup, a DROP DATABASE CASCADE will be ran after the test.
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 7
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Tue, 22 Nov 2022 15:56:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728 Set fallback database for functions .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11887/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 7 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 22 Nov 2022 13:57:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728 Set fallback database for functions .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11886/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 22 Nov 2022 13:53:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#7). Change subject: IMPALA-11728 Set fallback database for functions .. IMPALA-11728 Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_set_fallback_db_for_functions in test_udfs.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 66 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/7 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 7 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728 Set fallback database for functions .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/19252/6/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/19252/6/tests/query_test/test_udfs.py@677 PS6, Line 677: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 22 Nov 2022 13:34:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#6). Change subject: IMPALA-11728 Set fallback database for functions .. IMPALA-11728 Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_set_fallback_db_for_functions in test_udfs.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 66 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/6 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Xiaoqing Gao has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: IMPALA-11728 Set fallback database for functions
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/19252/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19252/4//COMMIT_MSG@7
PS4, Line 7: [IMPALA-11728]
> we generally use the IMPALA-11728: ... format.
Done
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java
File fe/src/main/java/org/apache/impala/analysis/FunctionName.java:
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@126
PS4, Line 126: preferBuiltinsDb
> I think that we should use fallback db only if preferBuiltinsDb is true - t
I checked that preferBuiltinsDb is false only in CreateFunctionStmtBase.java
and DropFunctionStmt.java. I modified the if statement.
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@128
PS4, Line 128: } else if (fallbackDbContainsFn(analyzer)) {
> I am not sure about the correct order here - checking the fallback db first
I'm not sure which function has higher priority, the fallback db or the
_impala_builtins? In the past, _impala_builtins have higher priority than
analyzer.getDefaultDb(). Maybe we can stay the same?
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@129
PS4, Line 129: analyzer.getQueryCtx()
: .getClient_request()
: .getQuery_options()
: .getFallback_db_for_functions();
> Can you create a function like String Analyzer.getFallbackDbForFunctions(),
Done
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py
File tests/custom_cluster/test_fallback_db_for_functions.py:
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py@28
PS4, Line 28: CustomClusterTestSuite
> Since there is a query option for fallback_db_for_functions, a simple (non
Done
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py@36
PS4, Line 36: fn
> It would be nice to also test a function name that also exists as builtin f
Done
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py@39
PS4, Line 39: functional_parquet
> It would be better to use a temporary db than persistent one like functiona
Thanks for your good suggestion. I changed the test.
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 4
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Tue, 22 Nov 2022 13:29:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728 Set fallback database for functions .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11885/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 22 Nov 2022 13:23:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Hello Quanlong Huang, lipeng...@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#5). Change subject: IMPALA-11728 Set fallback database for functions .. IMPALA-11728 Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_set_fallback_db_for_functions in test_udfs.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/query_test/test_udfs.py 7 files changed, 64 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/5 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11728 Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: IMPALA-11728 Set fallback database for functions .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/19252/5/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/19252/5/tests/query_test/test_udfs.py@34 PS5, Line 34: from tests.util.filesystem_utils import get_fs_path, IS_S3, WAREHOUSE flake8: F401 'tests.util.filesystem_utils.IS_S3' imported but unused http://gerrit.cloudera.org:8080/#/c/19252/5/tests/query_test/test_udfs.py@653 PS5, Line 653: E flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/19252/5/tests/query_test/test_udfs.py@669 PS5, Line 669: U flake8: E501 line too long (94 > 90 characters) http://gerrit.cloudera.org:8080/#/c/19252/5/tests/query_test/test_udfs.py@675 PS5, Line 675: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Tue, 22 Nov 2022 13:12:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/19252 )
Change subject: [IMPALA-11728] Set fallback database for functions
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/19252/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19252/4//COMMIT_MSG@7
PS4, Line 7: [IMPALA-11728]
we generally use the IMPALA-11728: ... format.
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java
File fe/src/main/java/org/apache/impala/analysis/FunctionName.java:
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@126
PS4, Line 126: preferBuiltinsDb
I think that we should use fallback db only if preferBuiltinsDb is true - there
are cases when we should ignore builtins, for example in CREATE FUNCTION / DROP
FUNCTION, and my impression is that preferBuiltinsDb is used in these cases (I
haven't checked all places where we call this function).
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@128
PS4, Line 128: } else if (fallbackDbContainsFn(analyzer)) {
I am not sure about the correct order here - checking the fallback db first
would allow a nice way to override builtins.
http://gerrit.cloudera.org:8080/#/c/19252/4/fe/src/main/java/org/apache/impala/analysis/FunctionName.java@129
PS4, Line 129: analyzer.getQueryCtx()
: .getClient_request()
: .getQuery_options()
: .getFallback_db_for_functions();
Can you create a function like String Analyzer.getFallbackDbForFunctions(),
similarly to getDefaultDb?
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py
File tests/custom_cluster/test_fallback_db_for_functions.py:
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py@28
PS4, Line 28: CustomClusterTestSuite
Since there is a query option for fallback_db_for_functions, a simple (non
CustomCluster) could be used, which would make test running faster.
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py@36
PS4, Line 36: fn
It would be nice to also test a function name that also exists as builtin
function and check what is happening.
http://gerrit.cloudera.org:8080/#/c/19252/4/tests/custom_cluster/test_fallback_db_for_functions.py@39
PS4, Line 39: functional_parquet
It would be better to use a temporary db than persistent one like
functional_parquet, as now the test is potentially dirty - for example if the
assert at line 47 fails, the function will not be dropped, which can affect
other tests.
I think that the simplest solution would be to move this test to an EE test,
for example in
https://github.com/apache/impala/blob/master/tests/query_test/test_udfs.py
In EE tests there is a fixture called unique_database, which can be used to get
a temporary database that will be cleaned up after the test. You can find
examples in test_udfs.py for using unique_database and for setting query
options.
--
To view, visit http://gerrit.cloudera.org:8080/19252
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5
Gerrit-Change-Number: 19252
Gerrit-PatchSet: 4
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Mon, 21 Nov 2022 14:03:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11876/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Mon, 21 Nov 2022 06:57:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Xiaoqing Gao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 4: It should be the second case. Users can create fallback db anytime. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Mon, 21 Nov 2022 06:41:01 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Hello lipeng...@sensorsdata.cn, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#4). Change subject: [IMPALA-11728] Set fallback database for functions .. [IMPALA-11728] Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_fallback_db_for_functions.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/FunctionName.java A tests/custom_cluster/test_fallback_db_for_functions.py 6 files changed, 81 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/4 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
lipeng...@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 3: One question here, where does the "fallback database" come from, is it created in HMS when impala is installed, or is it possible to create a 'tool database' at any time and just set the 'tool database' name to query option when querying? -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Nov 2022 03:58:10 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 14:09:16 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 09:18:44 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 2 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 09:14:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11862/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 1 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 09:13:41 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#3). Change subject: [IMPALA-11728] Set fallback database for functions .. [IMPALA-11728] Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_fallback_db_for_functions.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/FunctionName.java A tests/custom_cluster/test_fallback_db_for_functions.py 6 files changed, 84 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/3 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8814/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 09:00:31 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 2: (6 comments) http://gerrit.cloudera.org:8080/#/c/19252/2/tests/custom_cluster/test_fallback_db_for_functions.py File tests/custom_cluster/test_fallback_db_for_functions.py: http://gerrit.cloudera.org:8080/#/c/19252/2/tests/custom_cluster/test_fallback_db_for_functions.py@24 PS2, Line 24: from ImpalaService import ImpalaHiveServer2Service flake8: F401 'ImpalaService.ImpalaHiveServer2Service' imported but unused http://gerrit.cloudera.org:8080/#/c/19252/2/tests/custom_cluster/test_fallback_db_for_functions.py@27 PS2, Line 27: class TestFallbackDbForFunctions(CustomClusterTestSuite, HS2TestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/19252/2/tests/custom_cluster/test_fallback_db_for_functions.py@46 PS2, Line 46: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/19252/2/tests/custom_cluster/test_fallback_db_for_functions.py@49 PS2, Line 49: flake8: W292 no newline at end of file http://gerrit.cloudera.org:8080/#/c/19252/2/tests/metadata/test_ddl.py File tests/metadata/test_ddl.py: http://gerrit.cloudera.org:8080/#/c/19252/2/tests/metadata/test_ddl.py@20 PS2, Line 20: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/2/tests/metadata/test_ddl.py@20 PS2, Line 20: import pytest line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 2 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 08:55:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19252 ) Change subject: [IMPALA-11728] Set fallback database for functions .. Patch Set 1: (9 comments) http://gerrit.cloudera.org:8080/#/c/19252/1/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/19252/1/tests/common/impala_test_suite.py@857 PS1, Line 857: r flake8: F706 'return' outside function http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py File tests/custom_cluster/test_fallback_db_for_functions.py: http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py@24 PS1, Line 24: from ImpalaService import ImpalaHiveServer2Service flake8: F401 'ImpalaService.ImpalaHiveServer2Service' imported but unused http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py@27 PS1, Line 27: class TestFallbackDbForFunctions(CustomClusterTestSuite, HS2TestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py@45 PS1, Line 45: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py@45 PS1, Line 45: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py@46 PS1, Line 46: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/19252/1/tests/custom_cluster/test_fallback_db_for_functions.py@49 PS1, Line 49: flake8: W292 no newline at end of file http://gerrit.cloudera.org:8080/#/c/19252/1/tests/metadata/test_ddl.py File tests/metadata/test_ddl.py: http://gerrit.cloudera.org:8080/#/c/19252/1/tests/metadata/test_ddl.py@20 PS1, Line 20: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/19252/1/tests/metadata/test_ddl.py@20 PS1, Line 20: import pytest line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 1 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 17 Nov 2022 08:53:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Xiaoqing Gao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19252 Change subject: [IMPALA-11728] Set fallback database for functions .. [IMPALA-11728] Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_fallback_db_for_functions.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M tests/common/impala_test_suite.py A tests/custom_cluster/test_fallback_db_for_functions.py M tests/metadata/test_ddl.py 8 files changed, 85 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/1 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 1 Gerrit-Owner: Xiaoqing Gao
[Impala-ASF-CR] [IMPALA-11728] Set fallback database for functions
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19252 to look at the new patch set (#2). Change subject: [IMPALA-11728] Set fallback database for functions .. [IMPALA-11728] Set fallback database for functions The create function statement can only works on specified database. When using the function in another database, it will throw an AnalysisException because unknown for database. Add a fallback database for functions as query option. It works on all database without changing query. Testing: - add test_fallback_db_for_functions.py Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/FunctionName.java A tests/custom_cluster/test_fallback_db_for_functions.py M tests/metadata/test_ddl.py 7 files changed, 84 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19252/2 -- To view, visit http://gerrit.cloudera.org:8080/19252 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37b7e126718fea1c50723cacbaed898b20bb55e5 Gerrit-Change-Number: 19252 Gerrit-PatchSet: 2 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Impala Public Jenkins
