[Impala-ASF-CR] IMPALA-12631: Improve count star performance for parquet scans
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20804 ) Change subject: IMPALA-12631: Improve count star performance for parquet scans .. Patch Set 3: > Since this will be a behavior change, do you mind adding a backend flag to > control this? Default to count using FileMetaData.num_rows, but back to > RowGroups.num_rows when flag is disabled. This way, user can revert to old > behavior if they do hit an inaccurate FileMetaData.num_rows issue. > > Basic performance benchmark is also desirable to ensure no regression happen > like IMPALA-11123. Maybe you can steal TPCDS-Q_COUNT_OPTIMIZED, > TPCDS-Q_COUNT_UNOPTIMIZED, and TPCDS-Q_COUNT_ZERO_SLOT from > https://gerrit.cloudera.org/c/19927 and run single_node_perf_run.py such as: > > ./bin/single_node_perf_run.py --num_impalads=3 \ > --workloads=tpcds --iterations=9 --table_formats=parquet/none/none \ > > --query_names=TPCDS-Q_COUNT_OPTIMIZED,TPCDS-Q_COUNT_UNOPTIMIZED,TPCDS-Q_COUNT_ZERO_SLOT > \ > asf-master > > Even better if you can do it with larger scale TPC-DS like 10GB: > > ./bin/single_node_perf_run.py --num_impalads=3 --load --scale=10 \ > --workloads=tpcds --iterations=9 --table_formats=parquet/none/none \ > > --query_names=TPCDS-Q_COUNT_OPTIMIZED,TPCDS-Q_COUNT_UNOPTIMIZED,TPCDS-Q_COUNT_ZERO_SLOT > \ > asf-master > > Using tpch_parquet.lineitem should be fine as well. Thanks for the guidance! I'll try to add a backend flag and do some performance tests. -- To view, visit http://gerrit.cloudera.org:8080/20804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9cd2448fe51a420d4559d0cc861c4d30822f4fd Gerrit-Change-Number: 20804 Gerrit-PatchSet: 3 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Tue, 19 Dec 2023 09:33:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12502: Support Impala to Impala federation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20731 ) Change subject: IMPALA-12502: Support Impala to Impala federation .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f Gerrit-Change-Number: 20731 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 19 Dec 2023 10:22:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10076/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 11:25:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Tamas Mate has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns As the slots have already been created on the frontend this change focuses on populating them on the backend side. There are two major parts of this commit. Obtaining the right Accessors for the slot and recursively filling the tuples with data. The field ids are present in the struct slot's ColumnType field as a list of integers. This list can be indexed with the correct element of the SchemaPath to obtain the field id for a struct member and with that the Accessor. Once the Accessors are available the IcebergRowReader's MaterializeTuple method can be called recursively to write the primitive slots of a struct slot. Testing: - Added E2E tests Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 --- M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 10 files changed, 280 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/20759/10 -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14792/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 11:53:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10077/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 12 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 12:33:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/20753/8/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/20753/8/tests/query_test/test_iceberg.py@1264 PS8, Line 1264: : @SkipIfDockerizedCluster.internal_hostname : @SkipIf.hardcoded_uris : def test_multiple_equality_ids(self, unique_database): : """This test loads an Iceberg table that has 2 equality delete files with different : equality ID lists. A query on such a table fails due to lack of support.""" : SRC_DIR = os.path.join(os.environ['IMPALA_HOME'], : "testdata/data/iceberg_test/hadoop_catalog/ice/" : "iceberg_v2_delete_different_equality_ids") : DST_DIR = "/test-warehouse/iceberg_test/hadoop_catalog/ice/" \ : "iceberg_v2_delete_different_equality_ids" > I see, thanks for giving it a try. I think this would add some extra complexity for this test but the gain wouldn't be that much because this is an error-case anyway. -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 11 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 12:33:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 13:11:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 12: Code-Review+2 (1 comment) Nice change! Thanks Gabor for working on this! http://gerrit.cloudera.org:8080/#/c/20753/8/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/20753/8/tests/query_test/test_iceberg.py@1264 PS8, Line 1264: @SkipIfDockerizedCluster.internal_hostname : @SkipIf.hardcoded_uris : def test_multiple_equality_ids(self, unique_database): : """This test loads an Iceberg table that has 2 equality delete files with different : equality ID lists. A query on such a table fails due to lack of support.""" : SRC_DIR = os.path.join(os.environ['IMPALA_HOME'], : "testdata/data/iceberg_test/hadoop_catalog/ice/" : "iceberg_v2_delete_different_equality_ids") : DST_DIR = "/test-warehouse/iceberg_test/hadoop_catalog/ice/" \ : "iceberg_v2_delete_different_equality_ids" : TBL_NAME = "iceberg_v2_delete_different_equality_ids" > I think this would add some extra complexity for this test but the gain wou Ack. -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 12 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 13:42:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20817 Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. IMPALA-11921: Give more time to test_large_sql in slow builds The test was flaky on ASAN builds, hopefully giving more time will solve this. Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 --- M tests/shell/test_shell_commandline.py 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/20817/1 -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20818 ) Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/737/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 15:02:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20818 ) Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/738/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 15:02:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20818 Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. IMPALA-12653: Update documentation about the UPDATE statement This patch adds documentation about the UPDATE statement. Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 --- M docs/impala_keydefs.ditamap M docs/topics/impala_iceberg.xml 2 files changed, 25 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/20818/1 -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20818 to look at the new patch set (#2). Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. IMPALA-12653: Update documentation about the UPDATE statement This patch adds documentation about the UPDATE statement. Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 --- M docs/impala_keydefs.ditamap M docs/topics/impala_iceberg.xml 2 files changed, 25 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/20818/2 -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20818 ) Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/737/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 15:07:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20817 ) Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14793/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 15:09:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20818 ) Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/738/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 15:10:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/20818 ) Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. Patch Set 2: (2 comments) Nice and clear. http://gerrit.cloudera.org:8080/#/c/20818/2/docs/topics/impala_iceberg.xml File docs/topics/impala_iceberg.xml: http://gerrit.cloudera.org:8080/#/c/20818/2/docs/topics/impala_iceberg.xml@506 PS2, Line 506: don't doesn't http://gerrit.cloudera.org:8080/#/c/20818/2/docs/topics/impala_iceberg.xml@512 PS2, Line 512: Only the merge-on-read update mode is supported nit: If the list items are full sentences, it is nicer to end each with a period. -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 19 Dec 2023 15:49:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10076/ -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 15:54:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10078/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 16:00:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 11: Forgot to update the planner test, with the new exception. -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 15:59:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Tamas Mate has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns As the slots have already been created on the frontend this change focuses on populating them on the backend side. There are two major parts of this commit. Obtaining the right Accessors for the slot and recursively filling the tuples with data. The field ids are present in the struct slot's ColumnType field as a list of integers. This list can be indexed with the correct element of the SchemaPath to obtain the field id for a struct member and with that the Accessor. Once the Accessors are available the IcebergRowReader's MaterializeTuple method can be called recursively to write the primitive slots of a struct slot. Testing: - Added E2E tests Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 --- M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 11 files changed, 280 insertions(+), 94 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/20759/11 -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14794/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 16:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10079/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 16:59:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 12: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10077/ -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 12 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 17:00:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20817 ) Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Dec 2023 16:58:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12502: Support Impala to Impala federation
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20731 ) Change subject: IMPALA-12502: Support Impala to Impala federation .. Patch Set 14: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/20731/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20731/14//COMMIT_MSG@10 PS14, Line 10: through JDBC external data source. Maybe also mention the new query profile counter in the commit message. http://gerrit.cloudera.org:8080/#/c/20731/14//COMMIT_MSG@11 PS14, Line 11: Setting query options for Impala will be supported in a follow patch. nit: follow -> following -- To view, visit http://gerrit.cloudera.org:8080/20731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f Gerrit-Change-Number: 20731 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 19 Dec 2023 16:59:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20817 ) Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Dec 2023 17:19:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20817 ) Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10080/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Dec 2023 17:19:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12593: Split test drop partition test table
Csaba Ringhofer has removed a vote on this change. Change subject: IMPALA-12593: Split test_drop_partition test table .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ifc080c6bde091e17d20adc95da998e36c484f768 Gerrit-Change-Number: 20746 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12593: Split test drop partition test table
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20746 ) Change subject: IMPALA-12593: Split test_drop_partition test table .. IMPALA-12593: Split test_drop_partition test table TestIcebergTable.test_drop_partition fails on the S3 builds because of too long file prefixes. This patch changes the test by creating multiple tables instead, and add more checkpointing to validate the drop operation. Change-Id: Ifc080c6bde091e17d20adc95da998e36c484f768 Reviewed-on: http://gerrit.cloudera.org:8080/20746 Reviewed-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer Tested-by: Csaba Ringhofer --- M testdata/workloads/functional-query/queries/QueryTest/iceberg-drop-partition.test M tests/query_test/test_iceberg.py 2 files changed, 273 insertions(+), 192 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Csaba Ringhofer: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifc080c6bde091e17d20adc95da998e36c484f768 Gerrit-Change-Number: 20746 Gerrit-PatchSet: 3 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12593: Split test drop partition test table
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20746 ) Change subject: IMPALA-12593: Split test_drop_partition test table .. Patch Set 2: Verified+1 Code-Review+2 the failed test seems to be an unrelated flaky one: test_mixed_catalog_ddls_with_invalidate_metadata see IMPALA-10363 Merging this one manually to make non HDFS builds green -- To view, visit http://gerrit.cloudera.org:8080/20746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc080c6bde091e17d20adc95da998e36c484f768 Gerrit-Change-Number: 20746 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 17:26:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12502: Support Impala to Impala federation
Wenzhe Zhou has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/20731 ) Change subject: IMPALA-12502: Support Impala to Impala federation .. IMPALA-12502: Support Impala to Impala federation This patch adds support to read Impala tables in the Impala cluster through JDBC external data source. It also adds a new counter NumExternalDataSourceGetNext in profile for the total number of calls to ExternalDataSource::GetNext(). Setting query options for Impala will be supported in a following patch. Testing: - Added an end-to-end unit test to read Impala tables from Impala cluster through JDBC external data source. Manually ran the unit-test with Impala tables in Impala cluster on a remote host by setting $INTERNAL_LISTEN_HOST in jdbc.url as the ip address of the remote host on which an Impala cluster is running. - Added LDAP test for reading table through JDBC external data source with LDAP authentication. Manually ran the unit-test with Impala tables in a remote Impala cluster. - Passed core tests. Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f --- M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/DatabaseType.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfig.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessorFactory.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java A java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/ImpalaDatabaseAccessor.java A testdata/bin/download-impala-jdbc-driver.sh A testdata/bin/remove-impala-jdbc-driver.sh A testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M tests/custom_cluster/test_ext_data_sources.py 13 files changed, 614 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/20731/15 -- To view, visit http://gerrit.cloudera.org:8080/20731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f Gerrit-Change-Number: 20731 Gerrit-PatchSet: 15 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 13 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 17:42:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10081/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 13 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 17:42:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12502: Support Impala to Impala federation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20731 ) Change subject: IMPALA-12502: Support Impala to Impala federation .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14795/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f Gerrit-Change-Number: 20731 Gerrit-PatchSet: 15 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 19 Dec 2023 18:04:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12653: Update documentation about the UPDATE statement
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20818 ) Change subject: IMPALA-12653: Update documentation about the UPDATE statement .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20818/2/docs/topics/impala_iceberg.xml File docs/topics/impala_iceberg.xml: http://gerrit.cloudera.org:8080/#/c/20818/2/docs/topics/impala_iceberg.xml@515 PS2, Line 515: Can only write data and delete files in Parquet format So if the table is ORC then we write Parquet delete files? If so then a few more details might be helpful. -- To view, visit http://gerrit.cloudera.org:8080/20818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a4f3dcdba5faaa7dffda60b8590d09e6a92a165 Gerrit-Change-Number: 20818 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 19 Dec 2023 18:03:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/20799/11/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/20799/11/be/src/service/impala-server.cc@1631 PS11, Line 1631:<< " closed before in-flight."; There seems to be another scenario besides retries where this can happen. Usually associated with errors earlier in the execution process via HS2. I'm going to limit this check to CloseClientRequestState called from retries, because that's the only case where we still except SetQueryInflight to be called. -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 18:28:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 11: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10079/ -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 18:38:25 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/20782/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20782/4//COMMIT_MSG@9 PS4, Line 9: The Impala website at ASF https://impala.apache.org/ is the first hit Extra space between the link and is? http://gerrit.cloudera.org:8080/#/c/20782/4//COMMIT_MSG@11 PS4, Line 11: to say ="Apache Impala is a modern, open source, distributed SQL query Why is there an '=' sign? -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 18:39:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20782 to look at the new patch set (#5). Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. IMPALA-12619: Update Impala website to reduce emphasis on Hadoop The Impala website at ASF https://impala.apache.org/ is the first hit returned for “Apache Impala”. Update the first line of the description to say "Apache Impala is a modern, open source, distributed SQL query engine for open data and table formats." instead of "Apache Impala is a modern, open source, distributed SQL query engine for Apache Hadoop." Also mention Ranger instead of Sentry, and add references to Iceberg. Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd --- M index.html 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/20782/5 -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 4: (2 comments) Thanks for the review http://gerrit.cloudera.org:8080/#/c/20782/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20782/4//COMMIT_MSG@9 PS4, Line 9: The Impala website at ASF https://impala.apache.org/ is the first hit > Extra space between the link and is? Done http://gerrit.cloudera.org:8080/#/c/20782/4//COMMIT_MSG@11 PS4, Line 11: to say ="Apache Impala is a modern, open source, distributed SQL query > Why is there an '=' sign? Done -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 18:48:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12502: Support Impala to Impala federation
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/20731 )
Change subject: IMPALA-12502: Support Impala to Impala federation
..
Patch Set 15:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20731/15/fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
File fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java:
http://gerrit.cloudera.org:8080/#/c/20731/15/fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java@737
PS15, Line 737: String downloadImpalaJdbcDriver = new
File(System.getenv("IMPALA_HOME"),
ImpalaJdbcClient uses HiveJdbc. Any reason we can't use that?
http://gerrit.cloudera.org:8080/#/c/20731/15/testdata/bin/download-impala-jdbc-driver.sh
File testdata/bin/download-impala-jdbc-driver.sh:
http://gerrit.cloudera.org:8080/#/c/20731/15/testdata/bin/download-impala-jdbc-driver.sh@39
PS15, Line 39: wget
"https://downloads.cloudera.com/connectors/ClouderaImpala_JDBC42-2.6.32.1041.zip";
The version to download would make sense as an environment variable in
impala-config.sh.
Also we should cache this locally so we don't need to download it every time.
http://gerrit.cloudera.org:8080/#/c/20731/15/tests/custom_cluster/test_ext_data_sources.py
File tests/custom_cluster/test_ext_data_sources.py:
http://gerrit.cloudera.org:8080/#/c/20731/15/tests/custom_cluster/test_ext_data_sources.py@125
PS15, Line 125: os.environ['IMPALA_HOME'],
'testdata/bin/download-impala-jdbc-driver.sh')
This setup seems strange. We already have ImpalaJdbcClient.java which produces
a runnable jar (via run-jdbc-client.sh). De we specifically need this driver?
--
To view, visit http://gerrit.cloudera.org:8080/20731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f
Gerrit-Change-Number: 20731
Gerrit-PatchSet: 15
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 19 Dec 2023 18:51:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 18:52:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10082/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 19:09:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Hello Quanlong Huang, Andrew Sherman, Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20799 to look at the new patch set (#12). Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. IMPALA-12633: Remove DCHECK for slow SetQueryInflight Removes the DCHECK that the original query is inflight before trying to close it during a query retry. SetQueryInflight is a separate operation the server performs after a query has started executing async, and it's possible for the query to fail and retry before the server calls SetQueryInflight. When that happens, we still need to perform cleanup or the original request_state is never closed and we hit a different DCHECK: "BlockOnWait() needs to be called!" Adds an option to CloseClientRequestState for when we close a ClientRequestState but the query is still in-flight with a new ClientRequestState, like for retries. It ensures that we bypass most of SetQueryInflight in case CloseClientRequestState was called first. Updates the message from DCHECK in ClientRequestState's destructor to reflect that wait_thread_ is only reset in Finalize. Adds a debug action and test where just the original query is delayed during the SetQueryInflight call. Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 --- M be/src/runtime/query-driver.cc M be/src/service/client-request-state.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M tests/custom_cluster/test_query_retries.py 5 files changed, 104 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/20799/12 -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/383/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 19:22:41 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-docs-submit/383/ -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 19:28:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14796/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 19:35:18 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10083/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 19:49:01 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10083/ -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 19:52:19 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Michael Smith has removed a vote on this change. Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: deleteVote Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12502: Support Impala to Impala federation
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20731 ) Change subject: IMPALA-12502: Support Impala to Impala federation .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/20731/15/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/20731/15/tests/custom_cluster/test_ext_data_sources.py@122 PS15, Line 122: donwload Nit: download -- To view, visit http://gerrit.cloudera.org:8080/20731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79ad3273932b658cb85c9c17cc834fa1b5fbd64f Gerrit-Change-Number: 20731 Gerrit-PatchSet: 15 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 19 Dec 2023 20:26:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: I don't think there's a Jenkins job to verify the site. It's static (except the blog, which uses a static generator but you're not changing it). I think best we have is locally inspect it. -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 20:27:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 20:31:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11805: Use llvm ObjectCache for codegen caching
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20733 ) Change subject: IMPALA-11805: Use llvm ObjectCache for codegen caching .. IMPALA-11805: Use llvm ObjectCache for codegen caching Currently, we employ llvm::ExecutionEngine for codegen caching, providing access to compiled functions within the cached engine. However, the real challenge is the ExecutionEngine uses a lot of memory which largely exceeds our memory estimates and it is very hard to predict. This patch addresses this issue by using llvm::ObjectCache for codegen caching. In our case, each execution engine would have only one module, and after the compilation of the module, the compiled codegened functions of the module would be set to the execution engine, therefore functions could be used by Impala. During function compilation within the module, if an ObjectCache is set to the execution engine, the compiled codegened functions would be also written into the cache. This way, if we keep the cache, when revisiting the same module (fragment), we can efficiently reuse the specific ObjectCache, loading pre-compiled codegened functions and saving time. The tpch performance test indicates no significant regression compared to the previous use of ExecutionEngine. Post-change, the actual memory usage of each codegen caching entry is notably reduced. +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(1) | parquet / none / none | 0.22| -0.65% | 0.20 | -0.75% | +--+---+-++++ +--+--+---++-++++---++-+---+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +--+--+---++-++++---++-+---+ | TPCH(1) | TPCH-Q13 | parquet / none / none | 0.49 | 0.47| +2.80% | 5.32%| 5.07%| 10| +1.22% | 1.63| 1.19 | | TPCH(1) | TPCH-Q4 | parquet / none / none | 0.16 | 0.16| +3.51% | 1.32%| * 10.38% * | 10| +0.06% | 0.49| 1.06 | | TPCH(1) | TPCH-Q11 | parquet / none / none | 0.12 | 0.12| +1.39% | 2.27%| 2.24%| 10| +1.50% | 1.90| 1.37 | | TPCH(1) | TPCH-Q19 | parquet / none / none | 0.21 | 0.21| +1.56% | * 10.02% * | * 11.42% * | 10| +1.18% | 0.57| 0.32 | | TPCH(1) | TPCH-Q18 | parquet / none / none | 0.27 | 0.27| +1.71% | 6.46%| 1.29%| 10| -0.19% | -1.19 | 0.81 | | TPCH(1) | TPCH-Q6 | parquet / none / none | 0.11 | 0.11| +0.79% | 2.76%| 2.15%| 10| +0.10% | 1.46| 0.71 | | TPCH(1) | TPCH-Q3 | parquet / none / none | 0.26 | 0.26| +0.71% | 6.63%| 6.18%| 10| +0.04% | 0.49| 0.25 | | TPCH(1) | TPCH-Q17 | parquet / none / none | 0.17 | 0.17| +0.41% | * 14.66% * | * 13.01% * | 10| +0.05% | 0.40| 0.07 | | TPCH(1) | TPCH-Q14 | parquet / none / none | 0.16 | 0.16| +0.19% | 1.41%| 1.39%| 10| +0.25% | 1.46| 0.31 | | TPCH(1) | TPCH-Q20 | parquet / none / none | 0.17 | 0.17| +0.22% | 1.70%| 1.77%| 10| -0.05% | -0.40 | 0.28 | | TPCH(1) | TPCH-Q12 | parquet / none / none | 0.16 | 0.16| -0.27% | 0.54%| 1.46%| 10| +0.14% | 0.93| -0.54 | | TPCH(1) | TPCH-Q22 | parquet / none / none | 0.11 | 0.11| -0.38% | 0.81%| 2.06%| 10| +0.03% | 0.22| -0.54 | | TPCH(1) | TPCH-Q16 | parquet / none / none | 0.17 | 0.17| -0.38% | 0.67%| 1.58%| 10| -0.01% | -0.13 | -0.70 | | TPCH(1) | TPCH-Q8 | parquet / none / none | 0.27 | 0.27| -0.08% | 1.24%| 1.15%| 10| -0.33% | -1.37 | -0.15 | | TPCH(1) | TPCH-Q15 | parquet / none / none | 0.16 | 0.16| -1.18% | * 16.61% * | * 10.25% * | 10| +0.33% | 0.40| -0.19 | | TPCH(1) | TPCH-Q1 | parquet / none / none | 0.22 | 0.22| -1.67% | 1.62%| 7.45%| 10| +0.43% | 1.02| -0.70 | | TPCH(1) | TPCH-Q5 | parquet / none / none | 0.22 | 0.22| -0.98% | 0.22%|
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Hello Kurt Deschler, Abhishek Rawat, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20612 to look at the new patch set (#18). Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. IMPALA-3825: Delegate runtime filter aggregation to some executors IMPALA-4400 improve the runtime filter by aggregating runtime filters locally before sending filter update to the coordinator and sharing a single RuntimeFilterBank for all fragment instances in a query. However, local filter aggregation is still insufficient if the number of nodes in an impala cluster is large. For example, in a cluster of around 700 impalad backends, aggregation of 1 MB bloom filter updates in the coordinator can exceed more than 1 second. This patch aims to reduce coordinator load and speed up runtime filter aggregation by doing intermediate aggregation in a few designated impala backends before doing final aggregation and publishing in the coordinator. Query option MAX_NUM_FILTERS_AGGREGATED_PER_HOST is added to control this feature. Given N as the number of backend executors excluding the coordinator, the selected number of intermediate aggregators M = ceil(N / MAX_NUM_FILTERS_AGGREGATED_PER_HOST). Setting MAX_NUM_FILTERS_AGGREGATED_PER_HOST <= 1 will disable the intermediate aggregator feature. In the backend scheduler, M impalad will be selected randomly as the intermediate aggregator for that runtime filter. Information of this M selected impalad then passed from the scheduler to coordinator as a RuntimeFilterAggregatorInfoPB. The coordinator then converts the RuntimeFilterAggregatorInfoPB into a filter routing information TRuntimeFilterAggDesc that is piggy-backed in TRuntimeFilterSource. A new RPC endpoint named UpdateFilterFromRemote is added in data_stream_service.proto to handle filter updates from fellow impalad executor to the designated aggregator impalad. This RPC will merge filter updates into 'pending_remote_filter'. The intermediate aggregator will then combine 'pending_remote_filter' with 'pending_merge_filter' (from local aggregation) into 'result_filter' which is then sent to the coordinator. RuntimeFilterBank of the intermediate aggregator will wait for all remote filter updates for at least RUNTIME_FILTER_WAIT_TIME_MS. If RuntimeFilterBank is closing and RUNTIME_FILTER_WAIT_TIME_MS has passed, any incomplete filter will be marked as ALWAYS_TRUE and sent to the coordinator. This patch currently targets the bloom filter produced by partitioned join build only. Another kind of runtime filter is still efficient to aggregate in coordinator only, while the bloom filter from broadcast join only requires 1 valid filter update for publishing. test_runtime_filters.py is modified to clarify the exec_options dimension, test matrix constraints, and reduce pytest.skip() calls on each test. runtime_filters.test is also changed to use counter aggregation and assert on ExecSummary table so that they stay valid irrespective of the number of fragment instances. We benchmark the aggregation speed of 1 MB runtime filter aggregation on 20 executor nodes cluster with MT_DOP=36 that is instrumented to disable local aggregation, simulating 720 runtime filter updates. The speed is approximated as the duration between the earliest time a filter update is made and the time that the coordinator publishes the complete filter. The result is following: +-++ | num aggregator node | Aggregation speed (ms) | +-++ | 0 | 1296 | | 1 | 1229 | | 2 |608 | | 4 |329 | | 8 |205 | +-++ Testing: - Exercise MAX_NUM_FILTERS_AGGREGATED_PER_HOST in test_runtime_filters.py and query-options-test.cc - Add TestRuntimeFiltersLateRemoteUpdate. - Add custom_cluster/test_runtime_filter_aggregation.py. - Pass exhaustive tests. Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 --- M be/src/common/logging.h M be/src/runtime/coordinator.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter.cc M be/src/runtime/runtime-filter.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/network-util.h M be/src/util/runtime-profile-counters.h M comm
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 18: (3 comments) http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.cc File be/src/runtime/runtime-filter-bank.cc: http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.cc@722 PS10, Line 722: t entry_limit = query_stat > Runtime filter is most needed in big scan fragment, which most likely distr Added TestRuntimeFiltersLateRemoteUpdate in ps18 to exercise this code path. http://gerrit.cloudera.org:8080/#/c/20612/17/be/src/runtime/runtime-filter-bank.cc File be/src/runtime/runtime-filter-bank.cc: http://gerrit.cloudera.org:8080/#/c/20612/17/be/src/runtime/runtime-filter-bank.cc@753 PS17, Line 753: k> l(entr > Just found out that SetFilter can not be called multiple times. I'll submit Replaced this with DisableBloomFilter in ps18. http://gerrit.cloudera.org:8080/#/c/20612/18/be/src/runtime/runtime-filter-bank.cc File be/src/runtime/runtime-filter-bank.cc: http://gerrit.cloudera.org:8080/#/c/20612/18/be/src/runtime/runtime-filter-bank.cc@412 PS18, Line 412: CombinePeerAndLocalUpdates CombinePeerAndLocalUpdates is refactored to cover all possible states that can happen during merge of local_filter and remote_filter. Local aggregation must be complete before calling CombinePeerAndLocalUpdates. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 21:15:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12654: Add query option QUERY CPU COUNT DIVISOR
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20819 Change subject: IMPALA-12654: Add query option QUERY_CPU_COUNT_DIVISOR .. IMPALA-12654: Add query option QUERY_CPU_COUNT_DIVISOR IMPALA-11604 adds a hidden backend flag named query_cpu_count_divisor to allow oversubscribing CPU cores more than what is available in the executor group set. This patch adds a query option with the same name and function so that CPU core matching can be tuned for individual queries. The query option takes precedence over the flag. Testing: - Add test case in test_executor_groups.py and query-options-test.cc Change-Id: I34ab47bd67509a02790c3caedb3fde4d1b6eaa78 --- M be/src/service/query-option-parser.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/backend-gflag-util.cc M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 9 files changed, 88 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20819/1 -- To view, visit http://gerrit.cloudera.org:8080/20819 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I34ab47bd67509a02790c3caedb3fde4d1b6eaa78 Gerrit-Change-Number: 20819 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14797/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 21:35:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12654: Add query option QUERY CPU COUNT DIVISOR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20819 ) Change subject: IMPALA-12654: Add query option QUERY_CPU_COUNT_DIVISOR .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14798/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20819 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I34ab47bd67509a02790c3caedb3fde4d1b6eaa78 Gerrit-Change-Number: 20819 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 19 Dec 2023 21:46:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12380 WIP
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20809 )
Change subject: IMPALA-12380 WIP
..
Patch Set 2:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java:
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@44
PS2, Line 44: private static final Logger LOGGER =
LoggerFactory.getLogger(JdbcStorageConfigManager.class);
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@64
PS2, Line 64: public static String getPasswordFromProperties(Properties
properties, Function keyTransform)
line too long (110 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@65
PS2, Line 65: // throws IOException
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@71
PS2, Line 71: // In tez, when the job conf is copied there is a code path
in HiveInputFormat where all the table properties
line too long (115 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@72
PS2, Line 72: // are copied and the password is copied from the job
credentials, so its possible to have 2 of them set.
line too long (111 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@73
PS2, Line 73: // For now ignore this and print a warning message, we
should fix so that the above code is used instead.
line too long (111 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@74
PS2, Line 74: LOGGER.warn("Only one of " + CONFIG_PWD + ", " +
CONFIG_PWD_KEYSTORE + ", " + CONFIG_PWD_URI + " can be set");
line too long (116 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@76
PS2, Line 76: //"Only one of " + CONFIG_PWD + ", " +
CONFIG_PWD_KEYSTORE + ", " + CONFIG_PWD_URI + " can be set");
line too long (110 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@92
PS2, Line 92: // Should I include the uri in the exception? Suppressing
for now, since it may have sensitive info.
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@111
PS2, Line 111: public static String getPasswdFromKeystore(String keystore,
String key) throws IOException {
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java@124
PS2, Line 124: public static String getPasswdFromUri(String uriString) throws
IOException, URISyntaxException {
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/SecretSource.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/SecretSource.java:
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/SecretSource.java@26
PS2, Line 26: * The URI scheme is used to match an URI to an implementation
scheme. The implementations are discovered and loaded
line too long (116 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/SecretSource.java@27
PS2, Line 27: * using java service loader. Currently, there isn't a way to
initialize or reset a SecretSource after construction.
line too long (116 > 90)
http://gerrit.cloudera.org:8080/#/c/20809/2/java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/URISecretSource.java
File
java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/URISecretSource.java:
http://gerrit.cloudera.org:8080/#/c/20809/2/j
[Impala-ASF-CR] IMPALA-12380 WIP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20809 to look at the new patch set (#2). Change subject: IMPALA-12380 WIP .. IMPALA-12380 WIP WIP Added some hive source files Added function getPasswordFromProperties(). Build successful Change-Id: Iec83a9b6e00456f0a1bbee747bd752b2cf9bf238 --- M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java A java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/SecretSource.java A java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/secrets/URISecretSource.java M testdata/bin/copy-ext-data-sources.sh A testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-dbcp-password.test M tests/custom_cluster/test_ext_data_sources.py 7 files changed, 315 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/20809/2 -- To view, visit http://gerrit.cloudera.org:8080/20809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iec83a9b6e00456f0a1bbee747bd752b2cf9bf238 Gerrit-Change-Number: 20809 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20817 ) Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Dec 2023 22:00:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11921: Give more time to test large sql in slow builds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20817 ) Change subject: IMPALA-11921: Give more time to test_large_sql in slow builds .. IMPALA-11921: Give more time to test_large_sql in slow builds The test was flaky on ASAN builds, hopefully giving more time will solve this. Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Reviewed-on: http://gerrit.cloudera.org:8080/20817 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/shell/test_shell_commandline.py 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id1177bee2a0ad2a0d02f8923369645af02063be8 Gerrit-Change-Number: 20817 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Gerrit-Change-Number: 20753 Gerrit-PatchSet: 13 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Dec 2023 22:14:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12597: Basic Equality delete read support for Iceberg tables
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20753 ) Change subject: IMPALA-12597: Basic Equality delete read support for Iceberg tables .. IMPALA-12597: Basic Equality delete read support for Iceberg tables In general, applying equality deletes is similar to how position deletes are applied to data files: using a LEFT ANTI JOIN where the SCAN for the data rows is on the left side while the SCAN for the delete rows is on the right side of the JOIN. The difference is the virtual columns and the conjuncts being used. For equality deletes the data sequence number of a delete file has to be greater than the data sequence number of the data file being investigated. This information is added as a virtual column to the scans and a conjunct is created in the JOIN node to check the relation. The equality delete fields from the delete files are checked agains the respective columns of the data SCANS. This patch makes it possible for Impala to read Iceberg tables with basic equality delete files. The Iceberg spec gives great flexibility for engines for writing equality deletes, however in practice Flink, one of the engines that write EQ-deletes supports only a subset of the use cases. This patch focuses on reading the EQ-deletes written by Flink. The restrictions are the following: - All equality delete files in a table should have the same equality field ID list. - For partitioned Iceberg tables it is expected that the partition values are also written into the equality delete files. - Tables with equality deletes shouldn't have partition or schema evolution. - Floating point equality columns aren't supported. - If a malformed equality delete file doesn't have some of the equality field IDs then Parquet reader will fill those missing fields with NULLs. As a side effect this will drop the rows from the result where the corresponding data columns have a null value. See IMPALA-11388 epic Jira for more details. Testing: - Checked if the existing functional_parquet.iceberg_v2_delete_equality table can be read successfully. - Added new test tables so that E2E tests can validate correctness. Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Reviewed-on: http://gerrit.cloudera.org:8080/20753 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.h M common/thrift/CatalogObjects.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java A fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergEqualityDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/data/0-0-38a471ff-46f4-4350-85cc-2e7ba946b34c-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/data/0-0-38a471ff-46f4-4350-85cc-2e7ba946b34c-2.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/data/0-0-72709aba-fb15-4bd6-9758-5f39eb9bdcb7-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/data/0-0-72709aba-fb15-4bd6-9758-5f39eb9bdcb7-2.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/data/delete-074a9e19e61b766e-652a169e0001_800513971_data.0.parq A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/0cf1a310-d39c-4c6a-bfef-c3fe33cd0c25-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/0cf1a310-d39c-4c6a-bfef-c3fe33cd0c25-m1.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/3d36bf90-2625-4625-b09b-d4359b979df9-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/3d36bf90-2625-4625-b09b-d4359b979df9-m1.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/bb4b8c07-84e1-421a-bb6c-594f297d118e-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/snap-3802179086205335895-1-3d36bf90-2625-4625-b09b-d4359b979df9.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/metadata/snap-8985205515767142888-1-0cf1a310-d39c-4c6a-bfef-c3fe33c
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 18: (2 comments) http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/ImpalaService.thrift@849 PS18, Line 849: // (no change). Defaults to 0.5. Why update this text, but not RUNTIME_FILTER_CARDINALITY_REDUCTION_SCALE or MAX_NUM_FILTERS_AGGREGATED_PER_HOST? http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/Query.thrift@692 PS18, Line 692: 172: optional double runtime_filter_cardinality_reduction_scale = 1.0 Why are we renumbering these rather than moving runtime_filter_cardinality_reduction_scale to the end? I don't think it matters in this case, but seems like a good habit for the times it does. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 22:09:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12380 WIP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20809 ) Change subject: IMPALA-12380 WIP .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14799/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20809 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec83a9b6e00456f0a1bbee747bd752b2cf9bf238 Gerrit-Change-Number: 20809 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Dec 2023 22:15:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 18: (2 comments) Sorry, I should mention that ps18 was rebased on top of recent asf-master HEAD (commit b37a35aa, IMPALA-12018). IMPALA-12018 adds query option RUNTIME_FILTER_CARDINALITY_REDUCTION_SCALE. That is why MAX_NUM_FILTERS_AGGREGATED_PER_HOST is shifted after that in ps18. http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/ImpalaService.thrift@849 PS18, Line 849: // (no change). Defaults to 0.5. > Why update this text, but not RUNTIME_FILTER_CARDINALITY_REDUCTION_SCALE or This is carried from parent commit b37a35aa (IMPALA-12018). http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/Query.thrift@692 PS18, Line 692: 172: optional double runtime_filter_cardinality_reduction_scale = 1.0 > Why are we renumbering these rather than moving runtime_filter_cardinality_ This is carried from parent commit b37a35aa (IMPALA-12018). -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 22:17:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 5: (2 comments) Thanks Michael, so if there is no gerrit, do you know how I submit? http://gerrit.cloudera.org:8080/#/c/20782/5/index.html File index.html: http://gerrit.cloudera.org:8080/#/c/20782/5/index.html@118 PS5, Line 118: for Apache Hadoop. > Can we update this line too? It's the first thing that draws your eye. Yes, good catch, not sure how I missed that http://gerrit.cloudera.org:8080/#/c/20782/5/index.html@133 PS5, Line 133: Utilize the same file and data formats and metadata, security, and resource management frameworks as your Hadoop deployment—no redundant infrastructure or data conversion/duplication. > This whole bullet would be nice to rethink, but we could handle that as a s Yes, I don't like "User-verse" but trying to limit the scope for now -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 22:22:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 18: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/20612/18/common/thrift/ImpalaService.thrift@849 PS18, Line 849: // (no change). Defaults to 0.5. > This is carried from parent commit b37a35aa (IMPALA-12018). Oh right. Makes a lot more sense if I look at the whole diff. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 22:27:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20782 to look at the new patch set (#6). Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. IMPALA-12619: Update Impala website to reduce emphasis on Hadoop The Impala website at ASF https://impala.apache.org/ is the first hit returned for “Apache Impala”. Update the first line of the description to say "Apache Impala is a modern, open source, distributed SQL query engine for open data and table formats." instead of "Apache Impala is a modern, open source, distributed SQL query engine for Apache Hadoop." Also mention Ranger instead of Sentry, and add references to Iceberg. Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd --- M index.html 1 file changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/20782/6 -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 6: Verified+1 Code-Review+1 Looks good to me. I think the release process for asf-site updates is automatic. I don't actually recall how it works. -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 22:31:09 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 19 Dec 2023 22:36:23 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12619: Update Impala website to reduce emphasis on Hadoop
Andrew Sherman has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20782 ) Change subject: IMPALA-12619: Update Impala website to reduce emphasis on Hadoop .. IMPALA-12619: Update Impala website to reduce emphasis on Hadoop The Impala website at ASF https://impala.apache.org/ is the first hit returned for “Apache Impala”. Update the first line of the description to say "Apache Impala is a modern, open source, distributed SQL query engine for open data and table formats." instead of "Apache Impala is a modern, open source, distributed SQL query engine for Apache Hadoop." Also mention Ranger instead of Sentry, and add references to Iceberg. Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Reviewed-on: http://gerrit.cloudera.org:8080/20782 Reviewed-by: Michael Smith Tested-by: Michael Smith Reviewed-by: Andrew Sherman --- M index.html 1 file changed, 6 insertions(+), 6 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve; Verified Andrew Sherman: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20782 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: I2d63bbbc87375345eaf58989a59f704dbb9559fd Gerrit-Change-Number: 20782 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 12: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 22:37:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20799 ) Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Patch Set 12: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10082/ -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Dec 2023 23:38:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12633: Remove DCHECK for slow SetQueryInflight
Michael Smith has removed a vote on this change. Change subject: IMPALA-12633: Remove DCHECK for slow SetQueryInflight .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ic17a5e12d9db61cb19306270174518a8dfd281a7 Gerrit-Change-Number: 20799 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 18: Code-Review+2 (3 comments) just some nits for the tests, feel free to merge after resolving them http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py File tests/query_test/test_runtime_filters.py: http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py@486 PS18, Line 486: work nit: works http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py@503 PS18, Line 503: ' nit: +2 indentation http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py@517 PS18, Line 517: result.data[0] = '620' is this supposed to be an assert? -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 20 Dec 2023 07:12:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Dec 2023 07:29:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Hello Kurt Deschler, Abhishek Rawat, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20612 to look at the new patch set (#19). Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. IMPALA-3825: Delegate runtime filter aggregation to some executors IMPALA-4400 improve the runtime filter by aggregating runtime filters locally before sending filter update to the coordinator and sharing a single RuntimeFilterBank for all fragment instances in a query. However, local filter aggregation is still insufficient if the number of nodes in an impala cluster is large. For example, in a cluster of around 700 impalad backends, aggregation of 1 MB bloom filter updates in the coordinator can exceed more than 1 second. This patch aims to reduce coordinator load and speed up runtime filter aggregation by doing intermediate aggregation in a few designated impala backends before doing final aggregation and publishing in the coordinator. Query option MAX_NUM_FILTERS_AGGREGATED_PER_HOST is added to control this feature. Given N as the number of backend executors excluding the coordinator, the selected number of intermediate aggregators M = ceil(N / MAX_NUM_FILTERS_AGGREGATED_PER_HOST). Setting MAX_NUM_FILTERS_AGGREGATED_PER_HOST <= 1 will disable the intermediate aggregator feature. In the backend scheduler, M impalad will be selected randomly as the intermediate aggregator for that runtime filter. Information of this M selected impalad then passed from the scheduler to coordinator as a RuntimeFilterAggregatorInfoPB. The coordinator then converts the RuntimeFilterAggregatorInfoPB into a filter routing information TRuntimeFilterAggDesc that is piggy-backed in TRuntimeFilterSource. A new RPC endpoint named UpdateFilterFromRemote is added in data_stream_service.proto to handle filter updates from fellow impalad executor to the designated aggregator impalad. This RPC will merge filter updates into 'pending_remote_filter'. The intermediate aggregator will then combine 'pending_remote_filter' with 'pending_merge_filter' (from local aggregation) into 'result_filter' which is then sent to the coordinator. RuntimeFilterBank of the intermediate aggregator will wait for all remote filter updates for at least RUNTIME_FILTER_WAIT_TIME_MS. If RuntimeFilterBank is closing and RUNTIME_FILTER_WAIT_TIME_MS has passed, any incomplete filter will be marked as ALWAYS_TRUE and sent to the coordinator. This patch currently targets the bloom filter produced by partitioned join build only. Another kind of runtime filter is still efficient to aggregate in coordinator only, while the bloom filter from broadcast join only requires 1 valid filter update for publishing. test_runtime_filters.py is modified to clarify the exec_options dimension, test matrix constraints, and reduce pytest.skip() calls on each test. runtime_filters.test is also changed to use counter aggregation and assert on ExecSummary table so that they stay valid irrespective of the number of fragment instances. We benchmark the aggregation speed of 1 MB runtime filter aggregation on 20 executor nodes cluster with MT_DOP=36 that is instrumented to disable local aggregation, simulating 720 runtime filter updates. The speed is approximated as the duration between the earliest time a filter update is made and the time that the coordinator publishes the complete filter. The result is following: +-++ | num aggregator node | Aggregation speed (ms) | +-++ | 0 | 1296 | | 1 | 1229 | | 2 |608 | | 4 |329 | | 8 |205 | +-++ Testing: - Exercise MAX_NUM_FILTERS_AGGREGATED_PER_HOST in test_runtime_filters.py and query-options-test.cc - Add TestRuntimeFiltersLateRemoteUpdate. - Add custom_cluster/test_runtime_filter_aggregation.py. - Pass exhaustive tests. Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 --- M be/src/common/logging.h M be/src/runtime/coordinator.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter.cc M be/src/runtime/runtime-filter.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/network-util.h M be/src/util/runtime-profile-counters.h M comm
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 19: Code-Review+2 (3 comments) Thank you Csaba and Michael for your review! Carry +2. http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py File tests/query_test/test_runtime_filters.py: http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py@486 PS18, Line 486: work > nit: works Done http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py@503 PS18, Line 503: > nit: +2 indentation Done http://gerrit.cloudera.org:8080/#/c/20612/18/tests/query_test/test_runtime_filters.py@517 PS18, Line 517: assert result.data[0] > is this supposed to be an assert? Yes! Thanks for catching this. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 19 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 20 Dec 2023 07:33:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns
Tamas Mate has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20759 ) Change subject: IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns .. IMPALA-12205: Add support to STRUCT type Iceberg Metadata table columns As the slots have already been created on the frontend this change focuses on populating them on the backend side. There are two major parts of this commit. Obtaining the right Accessors for the slot and recursively filling the tuples with data. The field ids are present in the struct slot's ColumnType field as a list of integers. This list can be indexed with the correct element of the SchemaPath to obtain the field id for a struct member and with that the Accessor. Once the Accessors are available the IcebergRowReader's MaterializeTuple method can be called recursively to write the primitive slots of a struct slot. Testing: - Added E2E tests Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Reviewed-on: http://gerrit.cloudera.org:8080/20759 Tested-by: Impala Public Jenkins Reviewed-by: Gabor Kaszab --- M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 11 files changed, 280 insertions(+), 94 deletions(-) Approvals: Impala Public Jenkins: Verified Gabor Kaszab: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I953ad7253b270f2855bfcaee4ad023d1c4469273 Gerrit-Change-Number: 20759 Gerrit-PatchSet: 12 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-3825: Delegate runtime filter aggregation to some executors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 ) Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14800/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 19 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 20 Dec 2023 07:58:59 + Gerrit-HasComments: No
