[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); ... 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ ... This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Error messages will also include SQL-style comments about any UDFs used. For example: select aggfn(int_col) over (partition by int_col) from functional.alltypesagg Throws: Aggregate function 'default.aggfn(int_col) /* NATIVE UDF */' not supported with OVER clause. Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Reviewed-on: http://gerrit.cloudera.org:8080/16188 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M tests/query_test/test_udfs.py 7 files changed, 88 insertions(+), 21 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 9 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 07 Aug 2020 20:08:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6826/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 07 Aug 2020 15:24:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6247/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 07 Aug 2020 15:03:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 07 Aug 2020 15:02:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 7: Code-Review+2 Pre-commit failure was due to IMPALA-10054. Rebased on master. Carrying +2. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 07 Aug 2020 15:02:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Hello Qifan Chen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16188 to look at the new patch set (#7). Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); ... 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ ... This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Error messages will also include SQL-style comments about any UDFs used. For example: select aggfn(int_col) over (partition by int_col) from functional.alltypesagg Throws: Aggregate function 'default.aggfn(int_col) /* NATIVE UDF */' not supported with OVER clause. Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M tests/query_test/test_udfs.py 7 files changed, 88 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/7 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6207/ -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 01 Aug 2020 06:08:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6207/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 01 Aug 2020 00:59:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 01 Aug 2020 00:59:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 31 Jul 2020 21:41:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16188 )
Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being
used
..
Patch Set 5:
(2 comments)
> Did you run exhaustive tests? Would be good to do that just to be sure
> nothing else needs to be updated.
Ran exhaustive tests, everything passed.
http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@222
PS5, Line 222: if (fn_ != null && !fnName_.isBuiltin()) {
> We do have the ToSqlOptions that could maybe control this to hide it in err
yeah using ToSqlOptions makes sense, I briefly looked into it but it didn't
seem that straightforward because ToSql is called in so many places.
http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@491
PS5, Line 491: "User Defined Functions (UDFs)"
> I wonder if it worth the effort to make the key more explicit:
Yeah that would be nice because it makes it consistent with the info from
toSql, unfortunately the info about whether it is a native vs. java udf is only
available in the fn_ instance variable, which isn't set until the end of the
function.
it's probably do-able, but maybe not worth the effort since the same info is in
the explain plan already.
--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 30 Jul 2020 19:04:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 5: (1 comment) Looks good to me! http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java: http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@491 PS5, Line 491: "User Defined Functions (UDFs)" I wonder if it worth the effort to make the key more explicit: such as "JAVA UDF" and/or "Native UDF". -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 29 Jul 2020 17:19:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16188 )
Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being
used
..
Patch Set 5:
(1 comment)
Did you run exhaustive tests? Would be good to do that just to be sure nothing
else needs to be updated.
http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
File fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java:
http://gerrit.cloudera.org:8080/#/c/16188/5/fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java@222
PS5, Line 222: if (fn_ != null && !fnName_.isBuiltin()) {
We do have the ToSqlOptions that could maybe control this to hide it in error
messages, but I'm not sure that it's worth the effort to do that.
--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 23 Jul 2020 19:54:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6701/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 23 Jul 2020 17:25:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 5: hmm I thought core tests were passing, but I guess not. I fixed the test failures and re-ran core tests and they are clean now. One un-foreseen consequence of adding the SQL-style comments to FunctionCallExpr#toSqlImpl, is that they show up in error messages as well (see the example in the commit message). This might actually be kinda nice, because it makes the error message more informative, but wanted to specifically call this out because I never intended this to happen. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 23 Jul 2020 17:03:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16188 to look at the new patch set (#5). Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); ... 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ ... This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Error messages will also include SQL-style comments about any UDFs used. For example: select aggfn(int_col) over (partition by int_col) from functional.alltypesagg Throws: Aggregate function 'default.aggfn(int_col) /* NATIVE UDF */' not supported with OVER clause. Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M tests/query_test/test_udfs.py 7 files changed, 88 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/5 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6156/ -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 21:11:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6156/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 15:59:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 15:59:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Jul 2020 00:52:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6663/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:31:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6662/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:25:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16188 to look at the new patch set (#3). Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M tests/query_test/test_udfs.py 3 files changed, 59 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/3 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@625 PS2, Line 625: ; > flake8: E703 statement ends with a semicolon Done http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@632 PS2, Line 632: t > flake8: E501 line too long (92 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:05:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@625 PS2, Line 625: ; flake8: E703 statement ends with a semicolon http://gerrit.cloudera.org:8080/#/c/16188/2/tests/query_test/test_udfs.py@632 PS2, Line 632: t flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:03:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16188 to look at the new patch set (#2). Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower Testing: * Added tests to test_udfs.py * Ran core tests Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M tests/query_test/test_udfs.py 3 files changed, 58 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/2 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 1: (1 comment) Addressed comments. Added a test as well. http://gerrit.cloudera.org:8080/#/c/16188/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16188/1//COMMIT_MSG@33 PS1, Line 33: User Defined Functions (UDFs): default.hive_lower, default.hive_lower > Should we deduplicate this list? It could get pretty huge for bigger plans Done -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Jul 2020 21:03:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16188/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16188/1//COMMIT_MSG@33 PS1, Line 33: User Defined Functions (UDFs): default.hive_lower, default.hive_lower Should we deduplicate this list? It could get pretty huge for bigger plans -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 Jul 2020 18:09:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16188 ) Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6584/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 13 Jul 2020 23:59:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16188 Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being used .. IMPALA-9478: Profiles should indicate if custom UDFs are being used Adds a marker to runtime profiles and explain plans indicating if custom (e.g. non-built in) user-defined functions are being used. For explain plans, a SQL-style comment is added after any function call. For runtime profiles, a new Frontend entry called "User Defined Functions (UDFs)" lists out all UDFs analyzed during planning. Take the following example: create function hive_lower(string) returns string location '/test-warehouse/hive-exec.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower'; set explain_level=3; explain select * from functional.alltypes order by hive_lower(string_col); 01:SORT order by: default.hive_lower(string_col) /* JAVA UDF */ ASC materialized: default.hive_lower(string_col) /* JAVA UDF */ This shows up in the runtime profile as well. When the above query is actually run, the runtime profile includes the following entry: Frontend User Defined Functions (UDFs): default.hive_lower, default.hive_lower Testing: * Will add tests after confirming approach Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 --- M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java 2 files changed, 23 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/1 -- To view, visit http://gerrit.cloudera.org:8080/16188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345 Gerrit-Change-Number: 16188 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
