[Impala-ASF-CR] IMPALA-7969: Always admit trivial queries immediately
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19214 ) Change subject: IMPALA-7969: Always admit trivial queries immediately .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a729764e3055d7eb11900c96c82ff53eb261f91 Gerrit-Change-Number: 19214 Gerrit-PatchSet: 7 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 24 Jan 2023 01:00:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7969: Always admit trivial queries immediately
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19214 ) Change subject: IMPALA-7969: Always admit trivial queries immediately .. Patch Set 7: (5 comments) http://gerrit.cloudera.org:8080/#/c/19214/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19214/5//COMMIT_MSG@33 PS5, Line 33: runni > using -> running Done http://gerrit.cloudera.org:8080/#/c/19214/5//COMMIT_MSG@42 PS5, Line 42: could be extended in fu > 'might be able to extend in future' => Done http://gerrit.cloudera.org:8080/#/c/19214/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19214/6//COMMIT_MSG@31 PS6, Line 31: If the maximum parallelism is reached, : the admission controller would try to admit the trivial query : via normal process. > Would be good to add a test case for this scenario, if we don't have one al test_trivial_query_multi_runs in the ee test should cover the case, and the testcase ensures successful runs when max parallelism is reached. Added test_trivial_query_multi_runs_fallback to create an error case that fallback and blocked by a long query then timeout. http://gerrit.cloudera.org:8080/#/c/19214/5/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/19214/5/be/src/scheduling/admission-controller.h@666 PS5, Line 666: in > on -> in Done http://gerrit.cloudera.org:8080/#/c/19214/5/be/src/scheduling/admission-controller.h@979 PS5, Line 979: the quer > 'schedule' => 'query' Done -- To view, visit http://gerrit.cloudera.org:8080/19214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a729764e3055d7eb11900c96c82ff53eb261f91 Gerrit-Change-Number: 19214 Gerrit-PatchSet: 7 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 24 Jan 2023 00:44:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7969: Always admit trivial queries immediately
Yida Wu has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/19214 ) Change subject: IMPALA-7969: Always admit trivial queries immediately .. IMPALA-7969: Always admit trivial queries immediately The idea of trivial query is to allow certain queries to bypass the admission control, and therefore accelerating the query execution even when the server resource is at capacity. It could benefit the queries that require a fast response while consuming the minimum resources. This patch adds support for the trivial query detection and allows an immediate admission for the trivial query. We define the trivial query as a subset of the coordinator-only query, and returns no more than one row. The definition is as below: - Must have PLAN ROOT SINK as the root - Can contain UNION and EMPTYSET nodes only - Results can not be over one row Examples of a trivial query: - select 1; - select * from table limit 0; - select * from table limit 0 union all select 1; - select 1, (2 + 3); Also, we restrict the parallelism of execution of the trivial query, each resource pool can execute no more than three trivial queries at the same time. If the maximum parallelism is reached, the admission controller would try to admit the trivial query via normal process. More precisely, if the cluster is running with a global admission controller, the max parallelism for the trivial query is three per resource pool, but if there is no global admission controller, each coordinator would admit the trivial queries based on its own local variable, therefore, the max parallelism would be three per coordinator per resource pool in this case. As the first patch, we try to keep the trivial query as simple as possible, and it could be extended in future. Added query option enable_trivial_query_for_admission to control whether the trivial query policy is enabled. Tests: Passed exhaustive tests. Added test_trivial_query and test_trivial_query_low_mem. Change-Id: I2a729764e3055d7eb11900c96c82ff53eb261f91 --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/planner/Planner.java A fe/src/main/java/org/apache/impala/planner/TrivialQueryChecker.java M tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_session_expiration.py M tests/custom_cluster/test_shell_interactive.py 16 files changed, 444 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/19214/7 -- To view, visit http://gerrit.cloudera.org:8080/19214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a729764e3055d7eb11900c96c82ff53eb261f91 Gerrit-Change-Number: 19214 Gerrit-PatchSet: 7 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11857: Connect join build fragment to join in graphical plan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19437 ) Change subject: IMPALA-11857: Connect join build fragment to join in graphical plan .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12221/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80af977e5c5e869268d3ee68fafe541adadc239d Gerrit-Change-Number: 19437 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Jan 2023 00:42:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11857: Connect join build fragment to join in graphical plan
Kurt Deschler has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19437 Change subject: IMPALA-11857: Connect join build fragment to join in graphical plan .. IMPALA-11857: Connect join build fragment to join in graphical plan When support was added for join build sink, the plan JSON and plan rendering logic were not updated to handle the new sink type. As a result, the linkage from the join build fragment to it's corresponding join node were not expressed in the JSON and build fragments nodes were rendered as orphaned. This change adds a new JSON join_build_target field to join build fragments and connects the build fragment to the join with a green dashed line similar to the red dashed line used for data sender fragments. Also changed the SVG fill type to 'none' for exchange edges to avoid rendering a black triangle if the right child was an exchange as in the join build case. Testing: - Manual testing with TPCH queries and reviewing rendered plan diagrams Change-Id: I80af977e5c5e869268d3ee68fafe541adadc239d --- M be/src/service/impala-http-handler.cc M www/query_plan.tmpl 2 files changed, 11 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/19437/1 -- To view, visit http://gerrit.cloudera.org:8080/19437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I80af977e5c5e869268d3ee68fafe541adadc239d Gerrit-Change-Number: 19437 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler
[Impala-ASF-CR] IMPALA-11476: Support Ozone erasure coding
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19324 ) Change subject: IMPALA-11476: Support Ozone erasure coding .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12220/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 Gerrit-Change-Number: 19324 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 23 Jan 2023 23:36:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11855: Upgrade jetty to 9.4.47
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19436 ) Change subject: IMPALA-11855: Upgrade jetty to 9.4.47 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbc6d3ad40b63986137ea1b5c71b9af61bd9e637 Gerrit-Change-Number: 19436 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Jan 2023 23:28:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11476: Support Ozone erasure coding
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19324 to look at the new patch set (#11). Change subject: IMPALA-11476: Support Ozone erasure coding .. IMPALA-11476: Support Ozone erasure coding Adds support for identifying erasure coding policy with Ozone. Enables testing Ozone with erasure coding. Omits support for identifying erasure coding policy with the o3fs protocol as that protocol is effectively deprecated and its classes don't provide access to the ObjectStore. Test updates: - test_exclusive_coordinator_plan: Ozone+EC blocks are 768MB, which is larger than all tables in our test environment. Use tpch_parquet which we rely on having 3 files (by loading from snapshot in this case). - test_new_file_shorter: receives an EOFException when seeking with EC - test_local_read: erasure-coded-bytes-read is also tied to IMPALA-11697 - test_erasure_coding: Ozone doesn't report files as erasure-coded (HDDS-7603) Testing: - Passes core E2E and custom cluster tests with TARGET_FILESYSTEM=ozone and ERASURE_CODING=true. Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/ozone-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_coordinators.py M tests/query_test/test_io_metrics.py M tests/query_test/test_scanners.py 10 files changed, 103 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/19324/11 -- To view, visit http://gerrit.cloudera.org:8080/19324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 Gerrit-Change-Number: 19324 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11476: Support Ozone erasure coding
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19324 to look at the new patch set (#10). Change subject: IMPALA-11476: Support Ozone erasure coding .. IMPALA-11476: Support Ozone erasure coding Adds support for identifying erasure coding policy with Ozone. Enables testing Ozone with erasure coding. Test updates: - test_exclusive_coordinator_plan: Ozone+EC blocks are 768MB, which is larger than all tables in our test environment. Use tpch_parquet which we rely on having 3 files (by loading from snapshot in this case). - test_new_file_shorter: receives an EOFException when seeking with EC - test_local_read: erasure-coded-bytes-read is also tied to IMPALA-11697 - test_erasure_coding: Ozone doesn't report files as erasure-coded (HDDS-7603) Testing: - Passes core E2E and custom cluster tests with TARGET_FILESYSTEM=ozone and ERASURE_CODING=true. Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/ozone-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_coordinators.py M tests/query_test/test_io_metrics.py M tests/query_test/test_scanners.py 10 files changed, 103 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/19324/10 -- To view, visit http://gerrit.cloudera.org:8080/19324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 Gerrit-Change-Number: 19324 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11476: Support erasure coding policy
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19324 to look at the new patch set (#9). Change subject: IMPALA-11476: Support erasure coding policy .. IMPALA-11476: Support erasure coding policy Adds support for identifying erasure coding policy with Ozone. Enables testing Ozone with erasure coding. Test updates: - test_exclusive_coordinator_plan: Ozone+EC blocks are 768MB, which is larger than all tables in our test environment. Use tpch_parquet which we rely on having 3 files (by loading from snapshot in this case). - test_new_file_shorter: receives an EOFException when seeking with EC - test_local_read: erasure-coded-bytes-read is also tied to IMPALA-11697 - test_erasure_coding: Ozone doesn't report files as erasure-coded (HDDS-7603) Testing: - Passes core E2E and custom cluster tests with TARGET_FILESYSTEM=ozone and ERASURE_CODING=true. Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/ozone-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_coordinators.py M tests/query_test/test_io_metrics.py M tests/query_test/test_scanners.py 10 files changed, 103 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/19324/9 -- To view, visit http://gerrit.cloudera.org:8080/19324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1 Gerrit-Change-Number: 19324 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 ) Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba Gerrit-Change-Number: 19432 Gerrit-PatchSet: 2 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 23 Jan 2023 18:36:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19432 ) Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs .. IMPALA-11807: Rewrite iceberg metadata if not on hdfs Iceberg test tables are usually written on hdfs and the file paths start with "hdfs://localhost:20500/test-warehouse". Earlier we manually transformed the metadata so paths would start with "/test-warehouse" Since IMPALA-11821, testdata/bin/rewrite-iceberg-metadata.py supports not only a custom WAREHOUSE_LOCATION_PREFIX, but the ability to trim the beginning of the file paths. This commit modifies the data load, so metadata rewrite always executes if not on hdfs, even with empty WAREHOUSE_LOCATION_PREFIX. Testing: - Ran iceberg tests on ozone and S3 Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba Reviewed-on: http://gerrit.cloudera.org:8080/19432 Reviewed-by: Michael Smith Tested-by: Impala Public Jenkins --- M testdata/bin/load-test-warehouse-snapshot.sh 1 file changed, 5 insertions(+), 3 deletions(-) Approvals: Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba Gerrit-Change-Number: 19432 Gerrit-PatchSet: 3 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-11855: Upgrade jetty to 9.4.47
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19436 ) Change subject: IMPALA-11855: Upgrade jetty to 9.4.47 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8982/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbc6d3ad40b63986137ea1b5c71b9af61bd9e637 Gerrit-Change-Number: 19436 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Jan 2023 18:20:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11745: Add Hive's ESRI geospatial functions as builtins
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/19425 )
Change subject: IMPALA-11745: Add Hive's ESRI geospatial functions as builtins
..
Patch Set 13:
(20 comments)
http://gerrit.cloudera.org:8080/#/c/19425/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19425/13//COMMIT_MSG@18
PS13, Line 18: limitations
Could mention that some of the limitations are that Impala doesn't support
varargs and generics.
http://gerrit.cloudera.org:8080/#/c/19425/13//COMMIT_MSG@34
PS13, Line 34: 7
Is there a reason for the different upper limits (6 and 7)?
http://gerrit.cloudera.org:8080/#/c/19425/13/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/19425/13/be/src/common/global-flags.cc@379
PS13, Line 379: "Specifies which implementation of geospatial functions
should be included "
Could include in the description what the possible values are.
http://gerrit.cloudera.org:8080/#/c/19425/13/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:
http://gerrit.cloudera.org:8080/#/c/19425/13/be/src/util/backend-gflag-util.cc@341
PS13, Line 341: NONE
The values of the possible strings could be extracted to constants somewhere
and used here and in global-flags.cc.
Thrift now supports operator<< for enums, so we could also get the string
values from there.
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py
File common/function-registry/gen_geospatial_udf_wrappers.py:
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@25
PS13, Line 25:
A comment could describe what this file is used for and how it is used.
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@26
PS13, Line 26: '
If we used triple quotes around the string we wouldn't have to put '\n\' at the
end of each line.
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@28
PS13, Line 28: // Licensed under the Apache License, Version 2.0 (the
"License");\n\
Is there a reason why the wording here is a bit different from the licence
header in this source file?
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@46
PS13, Line 46: size
Nit: parameter number?
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@48
PS13, Line 48: WRAPPERS
We should document the structure of this list.
We could also use dictionaries or a class for the elements.
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@102
PS13, Line 102: order
Maybe 'num' would be better here and for 'generate_argument()'.
http://gerrit.cloudera.org:8080/#/c/19425/13/common/function-registry/gen_geospatial_udf_wrappers.py@167
PS13, Line 167: def generate_file(config):
The file we generate is not very readable and also contains whitespace at the
end of some lines.
We could make it a bit more readable by adding newlines in and between method
definitions etc.
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java
File fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java@155
PS13, Line 155: ,
Nit: full stop instead of comma.
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java@207
PS13, Line 207: minNumberOfArguments
If we multiply it with config.increment (on L210) then this is not really the
min number of arguments (and the same for the max number). We should come up
with a better name and also add a doc comment about this in WrapperConfig.
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java@208
PS13, Line 208: .limit(config.maxNumberOfArguments)
Why do we need this limit?
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java@248
PS13, Line 248: wkwrap
Wouldn't 'wkIdWrap' be more informative?
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java@261
PS13, Line 261: LOG.error("Invalid arguments - one or more arguments
are null.");
The error message doesn't seem to be correct - we get here if 'geomref' is null
or empty, the other parameter ('wkwrap') is not checked in the corresponding if
condition.
http://gerrit.cloudera.org:8080/#/c/19425/13/fe/src/main/java/org/apache/impala/hive/executor/BinaryToBinaryHiveLegacyFunctionExtractor.java
File
fe/src/main/java/or
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 ) Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8981/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba Gerrit-Change-Number: 19432 Gerrit-PatchSet: 2 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 23 Jan 2023 13:29:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 ) Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8980/ -- To view, visit http://gerrit.cloudera.org:8080/19432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba Gerrit-Change-Number: 19432 Gerrit-PatchSet: 2 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 23 Jan 2023 13:13:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3880: Add list of all tables queried to runtime profile
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19401 ) Change subject: IMPALA-3880: Add list of all tables queried to runtime profile .. Patch Set 3: Code-Review+1 Thanks, LGTM! -- To view, visit http://gerrit.cloudera.org:8080/19401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib474a5c6522032679701103aa225a18edca62f5a Gerrit-Change-Number: 19401 Gerrit-PatchSet: 3 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 23 Jan 2023 10:26:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11013 (part 1): Support 'MIGRATE TABLE' for external Hdfs tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19397 ) Change subject: IMPALA-11013 (part 1): Support 'MIGRATE TABLE' for external Hdfs tables .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/19397/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19397/6//COMMIT_MSG@35 PS6, Line 35: - Child query 4: Drop the temporary Hdfs table. > I read that in Spark "When you migrate a Hive table to Iceberg, a backup of That's a good idea. And if we cannot undo the changes automatically, then at least the error message should provide the necessary information how to fix things. -- To view, visit http://gerrit.cloudera.org:8080/19397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91e6a9cfe099c263f17b5506d6db459b79ad31a5 Gerrit-Change-Number: 19397 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Jan 2023 10:56:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11845: Fix incorrect check of struct STAR path in resolvePathWithMasking
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19429 ) Change subject: IMPALA-11845: Fix incorrect check of struct STAR path in resolvePathWithMasking .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/19429/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19429/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1196 PS5, Line 1196: resolvedPath.getMatchedTypes( > Is it true that when 'pathType' is PathType.STAR and resolvedPath.getMatche We could also add a preconditions check below that the matched types include a struct if we don't return here. -- To view, visit http://gerrit.cloudera.org:8080/19429 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f1e78e325baafbe23101909d47e82bf140a2d77 Gerrit-Change-Number: 19429 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Jan 2023 09:55:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 ) Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8980/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba Gerrit-Change-Number: 19432 Gerrit-PatchSet: 2 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 23 Jan 2023 08:06:33 + Gerrit-HasComments: No
