[Impala-ASF-CR] IMPALA-12823: Fix repeated query not found messages in impalad.INFO logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21052 ) Change subject: IMPALA-12823: Fix repeated query not found messages in impalad.INFO logs .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15274/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d994482b6f67a2d09c6a924eaaf90b230245b61 Gerrit-Change-Number: 21052 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 22 Feb 2024 07:59:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12823: Fix repeated query not found messages in impalad.INFO logs
Surya Hebbar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21052 Change subject: IMPALA-12823: Fix repeated query not found messages in impalad.INFO logs .. IMPALA-12823: Fix repeated query not found messages in impalad.INFO logs If an unknown or closed query id is requested from the impala server. The page repeatedly queries the server producing repeated query not found messages in the impalad.INFO logs. The behaviour is seen in query plan and query summary pages. This has been fixed by stopping further requests on receiving an error response from the server. Change-Id: I6d994482b6f67a2d09c6a924eaaf90b230245b61 --- M www/query_plan.tmpl M www/query_summary.tmpl 2 files changed, 9 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/21052/1 -- To view, visit http://gerrit.cloudera.org:8080/21052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6d994482b6f67a2d09c6a924eaaf90b230245b61 Gerrit-Change-Number: 21052 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG@16 PS5, Line 16: impala, nit: Postgres http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG@17 PS5, Line 17: impala nit: remote impala http://gerrit.cloudera.org:8080/#/c/21015/5//COMMIT_MSG@19 PS5, Line 19: timetamp nit: timestamp http://gerrit.cloudera.org:8080/#/c/21015/5/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/5/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@33 PS5, Line 33: import org.apache.impala.analysis.TimestampLiteral; nit: keep alphabet order http://gerrit.cloudera.org:8080/#/c/21015/5/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test File testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test: http://gerrit.cloudera.org:8080/#/c/21015/5/testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test@275 PS5, Line 275: between it's better to use count(*) for 'between', like select count(*) from alltypes_jdbc_datasource where timestamp_col between '2009-01-03 00:12:00' and '2009-01-05 00:12:00'. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 22 Feb 2024 06:44:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12833: Enabled 'catalogd ha reset metadata on failover' by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21051 ) Change subject: IMPALA-12833: Enabled 'catalogd_ha_reset_metadata_on_failover' by default .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15273/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc7c529f34b70734a700ac0d9d58b7e5b0215f8d Gerrit-Change-Number: 21051 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 22 Feb 2024 05:50:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 ) Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10290/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 22 Feb 2024 05:42:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12833: Enabled 'catalogd ha reset metadata on failover' by default
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21051 Change subject: IMPALA-12833: Enabled 'catalogd_ha_reset_metadata_on_failover' by default .. IMPALA-12833: Enabled 'catalogd_ha_reset_metadata_on_failover' by default This patch set the default value of catalog server starting flag 'catalogd_ha_reset_metadata_on_failover' as true so that catalogd will reset its metadata when it becomes active. Testing: - Looped to run unit-tests for catalog HA and statestore HA without failure. - Passed core tests Change-Id: Ibc7c529f34b70734a700ac0d9d58b7e5b0215f8d --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/21051/1 -- To view, visit http://gerrit.cloudera.org:8080/21051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibc7c529f34b70734a700ac0d9d58b7e5b0215f8d Gerrit-Change-Number: 21051 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 11: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10289/ -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Thu, 22 Feb 2024 04:26:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15272/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 22 Feb 2024 02:58:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
File fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java:
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@233
PS4, Line 233: String propertyValue =
> line too long (94 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@249
PS4, Line 249: dataSourceName, true);
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@276
PS4, Line 276: }
> line too long (95 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@287
PS4, Line 287: "stored by JDBC. Missing required metadata: %s",
name_, key));
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java:
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java@127
PS4, Line 127: throw new TableLoadingException(String.format("Failed to
load table %s " +
> line too long (91 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Thu, 22 Feb 2024 02:33:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has uploaded a new patch set (#5). (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
IMPALA-12793: Create JDBC table without data source
This patch changes syntax of creating JDBC table statement as
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
(col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
)
[COMMENT 'table_comment']
STORED BY JDBC
TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)
Both "STORED BY JDBC" and "STORED AS JDBC" are acceptable. A table
property '__IMPALA_DATA_SOURCE_NAME' is added to the JDBC table with
value 'impalajdbcdatasource', which is shown in the output of command
'show create table'.
Testing:
- Added frontend unit tests for new syntax.
- Updated end-to-end unit tests to create JDBC tables without
data source.
- Passed core tests
Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
---
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
A fe/src/main/java/org/apache/impala/util/JsonUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M testdata/bin/create-ext-data-source-table.sql
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M
testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test
M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
M
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/custom_cluster/test_ext_data_sources.py
M tests/query_test/test_ext_data_sources.py
26 files changed, 478 insertions(+), 406 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/21016/5
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-11123: Reimplement ORC optimized count star
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19927 ) Change subject: IMPALA-11123: Reimplement ORC optimized count star .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15271/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5971c8f278e1dee44e2a8dd4d2f043d22ebf5d17 Gerrit-Change-Number: 19927 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 22 Feb 2024 02:19:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11123: Reimplement ORC optimized count star
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19927 ) Change subject: IMPALA-11123: Reimplement ORC optimized count star .. Patch Set 3: ps3 simplify the patch. -- To view, visit http://gerrit.cloudera.org:8080/19927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5971c8f278e1dee44e2a8dd4d2f043d22ebf5d17 Gerrit-Change-Number: 19927 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 22 Feb 2024 01:54:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11123: Reimplement ORC optimized count star
Hello Quanlong Huang, David Rorke, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19927 to look at the new patch set (#3). Change subject: IMPALA-11123: Reimplement ORC optimized count star .. IMPALA-11123: Reimplement ORC optimized count star Commit 7ca20b3c94b1c9c1ddd4ed1e89f0969a0df55330 revert the original optimized count(star) for ORC scan from commit f932d78ad0a30e322d59fc39072f710f889d2135 (gerrit review http://gerrit.cloudera.org:8080/18327). The revert is necessary since the unification of count star and zero slot functions into HdfsColumnarScanner and causing significant regression for non-optimized counts star query in parquet format (over 15% slower MaterializeTupleTime). This patch reimplements optimized count(star) for ORC scan code path while minimizing the code changes needed for parquet scan code path. After this patch, ORC and parquet code path will have only the following new things in common: - THdfsScanNode.count_star_slot_offset renamed to THdfsScanNode.star_slot_offset - HdfsScanner::IssueFooterRanges will only issue footer ranges if IsZeroSlotTableScan() or optimize_count_star() is true (made possible for parquet by IMPALA-12631). The structure of HdfsParquetScanner::GetNextInternal() remains unchanged. Its zero scan slot code path is still served through num_rows metadata from the parquet footer, while the optimized count star code path still loops over row groups metadata (also from parquet footer). The following table shows single-node benchmark result of 3 count query variant on TPC-DS scale 10, both in ORC and parquet format, looped 9 times. +---+---+-++-++ | Workload | Query | Format | Avg(s) | Base Avg(s) | Delta(Avg) | +---+---+-++-++ | TPCDS(10) | TPCDS-Q_COUNT_UNOPTIMIZED | orc | 0.30 | 0.28| +6.50% | | TPCDS(10) | TPCDS-Q_COUNT_OPTIMIZED | parquet | 0.14 | 0.14| +1.56% | | TPCDS(10) | TPCDS-Q_COUNT_ZERO_SLOT | parquet | 0.27 | 0.27| +1.42% | | TPCDS(10) | TPCDS-Q_COUNT_ZERO_SLOT | orc | 0.28 | 0.29| -3.03% | | TPCDS(10) | TPCDS-Q_COUNT_UNOPTIMIZED | parquet | 0.21 | 0.22| -4.45% | | TPCDS(10) | TPCDS-Q_COUNT_OPTIMIZED | orc | 0.14 | 0.21| I -35.92% | +---+---+-++-++ Testing: - Restore PlannerTest.testOrcStatsAgg - Restore TestAggregationQueriesRunOnce and TestAggregationQueriesRunOnce::test_orc_count_star_optimization - Exercise count(star) in TestOrc::test_misaligned_orc_stripes - Pass core tests Change-Id: I5971c8f278e1dee44e2a8dd4d2f043d22ebf5d17 --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/exec/orc/hdfs-orc-scanner.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test A testdata/workloads/functional-query/queries/QueryTest/orc-stats-agg.test M testdata/workloads/functional-query/queries/QueryTest/partition-key-scans.test M testdata/workloads/functional-query/queries/QueryTest/scanners.test M tests/query_test/test_aggregation.py M tests/query_test/test_scanners.py M tests/util/test_file_parser.py 17 files changed, 768 insertions(+), 84 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/19927/3 -- To view, visit http://gerrit.cloudera.org:8080/19927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5971c8f278e1dee44e2a8dd4d2f043d22ebf5d17 Gerrit-Change-Number: 19927 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 ) Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15270/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 22 Feb 2024 00:16:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10289/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Wed, 21 Feb 2024 23:53:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Wed, 21 Feb 2024 23:53:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 10: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Wed, 21 Feb 2024 23:53:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 ) Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. Patch Set 5: > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10287/ Reverted changes in pretty-printer.h to fix the test failures. We can improve the timestamp string in a separate patch. -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 21 Feb 2024 23:51:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20986 to look at the new patch set (#5). Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. IMPALA-12782: Show info of the event processing in /events webUI The /events page of catalogd shows the metrics and status of the event-processor. This patch adds more info in this page, including - lag info - current event batch that's being processing See the screenshot attached in the JIRA for how it looks like. Also moves the error message to the top to highlight the error status. Adds a debug flag, event_processing_delay_s, to inject a sleep while processing an event. So the web page can be captured more easily. Tests: - Add e2e test to verify the content of the page. Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c --- M be/src/catalog/catalog-server.cc M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_web_pages.py M www/events.tmpl 10 files changed, 291 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20986/5 -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12825: Install thrift into the impala-python virtualenv
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21046 ) Change subject: IMPALA-12825: Install thrift into the impala-python virtualenv .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib36e8a1ce8d446b69b08e81ea458f95c158e28f5 Gerrit-Change-Number: 21046 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Feb 2024 23:31:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21047/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21047/3//COMMIT_MSG@9 PS3, Line 9: 408c606 nit. include the related Jira ticket IMPALA-12426 in the commit message might be beneficial. -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 21 Feb 2024 22:25:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15269/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 21 Feb 2024 22:12:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 21 Feb 2024 21:51:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15268/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 21 Feb 2024 21:54:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Hello Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21047 to look at the new patch set (#3). Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. IMPALA-12828: Remove Usage of Unnecessary this-> The commit 408c606 added code that used the pattern "this->" unnecessarily. Remove instances of using "this->" from the code changes in this commit. Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 --- M be/src/service/impala-server.cc M be/src/service/internal-server-test.cc M be/src/service/internal-server.cc 3 files changed, 59 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/21047/3 -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. Patch Set 3: > Patch Set 1: > > (1 comment) > > Looks like be/src/service/internal-server-test.cc can be clean up as well. Yes, I got that file as well in patch 2 -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 21 Feb 2024 21:46:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12828: Remove Usage of Unnecessary this->
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828: Remove Usage of Unnecessary this-> .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21047/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21047/1//COMMIT_MSG@7 PS1, Line 7: IMPALA-12828 > nit: missing colon after JIRA number. Done -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 21 Feb 2024 21:46:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12828 Remove Usage of Unnecessary this->
Hello Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21047 to look at the new patch set (#2). Change subject: IMPALA-12828 Remove Usage of Unnecessary this-> .. IMPALA-12828 Remove Usage of Unnecessary this-> The commit 408c606 added code that used the pattern "this->" unnecessarily. Remove instances of using "this->" from the code changes in this commit. Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 --- M be/src/service/impala-server.cc M be/src/service/internal-server-test.cc M be/src/service/internal-server.cc 3 files changed, 59 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/21047/2 -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12828 Remove Usage of Unnecessary this->
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21047 ) Change subject: IMPALA-12828 Remove Usage of Unnecessary this-> .. Patch Set 1: (1 comment) Looks like be/src/service/internal-server-test.cc can be clean up as well. http://gerrit.cloudera.org:8080/#/c/21047/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21047/1//COMMIT_MSG@7 PS1, Line 7: IMPALA-12828 nit: missing colon after JIRA number. -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 21 Feb 2024 21:41:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12828 Remove Usage of Unnecessary this->
Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21047 Change subject: IMPALA-12828 Remove Usage of Unnecessary this-> .. IMPALA-12828 Remove Usage of Unnecessary this-> The commit 408c606 added code that used the pattern "this->" unnecessarily. Remove instances of using "this->" from the code changes in this commit. Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 --- M be/src/service/impala-server.cc M be/src/service/internal-server.cc 2 files changed, 46 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/21047/1 -- To view, visit http://gerrit.cloudera.org:8080/21047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6b1d8ba7e27d20ba1ac83caff5e56fd0f9347c0 Gerrit-Change-Number: 21047 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr
[Impala-ASF-CR] IMPALA-12825: Install thrift into the impala-python virtualenv
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21046 ) Change subject: IMPALA-12825: Install thrift into the impala-python virtualenv .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15267/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib36e8a1ce8d446b69b08e81ea458f95c158e28f5 Gerrit-Change-Number: 21046 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Feb 2024 19:20:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12638: Dump symbols for libstdc++/libc during finalize.sh
Joe McDonnell has abandoned this change. ( http://gerrit.cloudera.org:8080/20800 ) Change subject: IMPALA-12638: Dump symbols for libstdc++/libc during finalize.sh .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/20800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I011e0c83f933bbbc78882dab66a021411c69c19d Gerrit-Change-Number: 20800 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12825: Install thrift into the impala-python virtualenv
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21046 Change subject: IMPALA-12825: Install thrift into the impala-python virtualenv .. IMPALA-12825: Install thrift into the impala-python virtualenv impala-python currently gets its Thrift from the toolchain by adding the appropriate Thrift toolchain directories to the PYTHONPATH. This is a problem when switching to Python 3, because the toolchain Thrift was built with Python 2 and this can produce complicated bugs. In general, it is also not a good idea to get Python dependencies from the toolchain. This switches to installing Thrift into the impala-python virtualenv, which lets the different Python versions have their own copy of compiled files. Testing: - Ran a core job Change-Id: Ib36e8a1ce8d446b69b08e81ea458f95c158e28f5 --- M README-build.md M bin/set-pythonpath.sh M infra/python/deps/requirements.txt 3 files changed, 2 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/21046/1 -- To view, visit http://gerrit.cloudera.org:8080/21046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib36e8a1ce8d446b69b08e81ea458f95c158e28f5 Gerrit-Change-Number: 21046 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15266/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 21 Feb 2024 18:05:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21015/4/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@148 PS4, Line 148: * Literals will finally be converted to string for pushdown. > It's probably okay to use string representation here as all Literals will a Thanks. Updated comment. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 21 Feb 2024 17:40:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#5). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for impala, mysql and impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timetamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 4 files changed, 450 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/5 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15265/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 21 Feb 2024 17:26:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has uploaded a new patch set (#4). (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
IMPALA-12793: Create JDBC table without data source
This patch changes syntax of creating JDBC table statement as
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
(col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
)
[COMMENT 'table_comment']
STORED BY JDBC
TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)
Both "STORED BY JDBC" and "STORED AS JDBC" are working.
Testing:
- Added frontend unit tests for new syntax.
- Updated end-to-end unit tests to create JDBC tables without
data source..
- Passed core tests
Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
---
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
A fe/src/main/java/org/apache/impala/util/JsonUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M testdata/bin/create-ext-data-source-table.sql
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M
testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test
M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
M
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/custom_cluster/test_ext_data_sources.py
M tests/query_test/test_ext_data_sources.py
26 files changed, 476 insertions(+), 406 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/21016/4
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
File fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java:
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@233
PS4, Line 233: String propertyValue = getTableProperty(msTbl,
config.getPropertyName(), null, false);
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@249
PS4, Line 249: dataSource_ = new TDataSource(dataSourceName, location,
className, apiVersionString);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@276
PS4, Line 276: private String
getTableProperty(org.apache.hadoop.hive.metastore.api.Table msTbl, String key,
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@287
PS4, Line 287: throw new TableLoadingException(String.format("Failed to
load table %s produced " +
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java:
http://gerrit.cloudera.org:8080/#/c/21016/4/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java@127
PS4, Line 127: throw new TableLoadingException(String.format("Failed to
load table %s produced " +
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Wed, 21 Feb 2024 17:02:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Hello Quanlong Huang, Abhishek Rawat, Yifan Zhang, Zihao Ye, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21020 to look at the new patch set (#10). Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. IMPALA-12801: Increase query_log_ default size and bound its memory. Coordinator's /queries page is useful to show information about recently run and completed queries. Having more entries will be helpful to inspect queries that completed further back. The maximum entry of this table is controlled by 'query_log_size' flag. Higher value means more queries to keep, but it also cost more memory overhead in coordinator. This patch increase 'query_log_size' default value from 100 to 200. This patch also add flag 'query_log_size_in_bytes' (default to 2GB) as an additional safeguard to evict entry from query_log_ when this limit exceeded, preventing query_log_ total memory to grow prohibitively large. 'query_log_size_in_bytes' is used in combination with 'query_log_size' to limit the number of QueryStateRecord to retain in query_log_, whichever is less. Testing: - Pass exhaustive tests. Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 --- M be/src/service/impala-server.cc M be/src/service/impala-server.h M docs/topics/impala_webui.xml M tests/custom_cluster/test_web_pages.py M www/queries.tmpl 5 files changed, 110 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21020/10 -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 10: (4 comments) http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1198 PS9, Line 1198: than > nit: "than" Done http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1208 PS9, Line 1208: cons > nit: use "const auto&". I think that's why the clang-tidy job failed. Done http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1217 PS9, Line 1217: cons > nit: const auto& Done http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1225 PS9, Line 1225: cons > nit: const auto& Done -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Wed, 21 Feb 2024 16:17:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 ) Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10287/ -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 21 Feb 2024 14:10:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Wenzhe Zhou has removed a vote on this change. Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Removed Code-Review+2 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-12609: Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21026 ) Change subject: IMPALA-12609: Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15264/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide10ccf10fc0abf5c270119ba7092c67e712ec49 Gerrit-Change-Number: 21026 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 21 Feb 2024 13:08:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12609: Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables
Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/21026 ) Change subject: IMPALA-12609: Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables .. IMPALA-12609: Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables After this change, the new SHOW METADATA TABLES IN statement can be used to list all the available metadata tables of an Iceberg table. Note that similarly to querying the contents of Iceberg metadata tables, this also requires fully qualified paths, e.g. SHOW METADATA TABLES IN functional_parquet.iceberg_query_metadata; works, but USE functional_parquet; SHOW METADATA TABLES IN iceberg_query_metadata; does not work. The available metadata tables for all Iceberg tables are the same, corresponding to the values of the enum "org.apache.iceberg.MetadataTableType", so there is actually no need to pass the name of the regular table for which the metadata table list is requested through Thrift. This change, however, does send the table name because this way - we can check also at the point of generating the list of metadata tables that the table is an Iceberg table - if we add support for metadata tables for other table formats, the table name/path will be necessary to determine the correct list of metadata tables Testing: - added and updated tests in ParserTest, AnalyzeDDLTest, ToSqlTest and AuthorizationStmtTest - added a custom cluster test in test_authorization.py - added functional tests in iceberg-metadata-tables.test Change-Id: Ide10ccf10fc0abf5c270119ba7092c67e712ec49 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/authorization/test_authorization.py 16 files changed, 323 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/21026/6 -- To view, visit http://gerrit.cloudera.org:8080/21026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ide10ccf10fc0abf5c270119ba7092c67e712ec49 Gerrit-Change-Number: 21026 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12771: Impala catalogd events-skipped may mark the wrong number
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21045 ) Change subject: IMPALA-12771: Impala catalogd events-skipped may mark the wrong number .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15263/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7aeb04e999b82187eb138c0b643ead259da22f1a Gerrit-Change-Number: 21045 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Feb 2024 12:35:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12771: Impala catalogd events-skipped may mark the wrong number
cclive1...@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21045 Change subject: IMPALA-12771: Impala catalogd events-skipped may mark the wrong number .. IMPALA-12771: Impala catalogd events-skipped may mark the wrong number The description of events-skipped metric is wrong. Some cases in Add partition event ,the metric will also be increased, besides for some other cases like alter partition the event is skipped and the log is printed but the events-skipped metric is not increased. Change-Id: I7aeb04e999b82187eb138c0b643ead259da22f1a --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 149 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/21045/1 -- To view, visit http://gerrit.cloudera.org:8080/21045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7aeb04e999b82187eb138c0b643ead259da22f1a Gerrit-Change-Number: 21045 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] IMPALA-12771: Impala catalogd events-skipped may mark the wrong number
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21045 ) Change subject: IMPALA-12771: Impala catalogd events-skipped may mark the wrong number .. Patch Set 1: (8 comments) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1056 PS1, Line 1056: metrics_.getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).inc(partitions.size()); line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1662 PS1, Line 1662: // both old table not removed and new table not add, that means we skip the rename event process line too long (102 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1663 PS1, Line 1663: // if one is true and the other is false, that means we need to use the function(removeTableIfNotAddedLater/addTableIfNotRemovedLater) line too long (140 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2633 PS1, Line 2633: int skippedNum = batchedEvents_.size() - eventsToProcess.size() - partitionEventsToForceReload.size(); line too long (108 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2635 PS1, Line 2635: metrics_.getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).inc(skippedNum); line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@235 PS1, Line 235: // in case of [CREATE|DROP|ALTER] events on [DATABASE|TABLE|PARTITION] which were ignored line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1986 PS1, Line 1986: // I think the reason why using >= is that when process create table event the table is IncompleteTable line too long (107 > 90) http://gerrit.cloudera.org:8080/#/c/21045/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@2034 PS1, Line 2034: // invalidate the table and the table will be IncompleteTable, then alter event process will skip; line too long (102 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7aeb04e999b82187eb138c0b643ead259da22f1a Gerrit-Change-Number: 21045 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Feb 2024 12:10:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12433: Share buffers among channels in KrpcDataStreamSender
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20719 ) Change subject: IMPALA-12433: Share buffers among channels in KrpcDataStreamSender .. IMPALA-12433: Share buffers among channels in KrpcDataStreamSender Before this patch each KrpcDataStreamSender::Channel had 2 OutboundRowBatch with its own serialization and compression buffers. This patch switches to use a single buffer per channel. This is enough to store the in-flight data in KRPC, while other buffers are only used during serialization and compression which is done for just a single channel at a time, so can be shared among channels. Memory estimates in the planner are not changed because the existing calculation has several issues (see IMPALA-12594). Change-Id: I64854a350a9dae8bf3af11c871882ea4750e60b3 Reviewed-on: http://gerrit.cloudera.org:8080/20719 Tested-by: Impala Public Jenkins Reviewed-by: Kurt Deschler Reviewed-by: Zihao Ye Reviewed-by: Michael Smith --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/outbound-row-batch.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java 8 files changed, 135 insertions(+), 96 deletions(-) Approvals: Impala Public Jenkins: Verified Kurt Deschler: Looks good to me, but someone else must approve Zihao Ye: Looks good to me, but someone else must approve Michael Smith: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I64854a350a9dae8bf3af11c871882ea4750e60b3 Gerrit-Change-Number: 20719 Gerrit-PatchSet: 16 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 ) Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10287/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 21 Feb 2024 09:38:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15262/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 21 Feb 2024 09:17:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 ) Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15261/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 21 Feb 2024 09:07:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 9: (4 comments) http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1198 PS9, Line 1198: that nit: "than" http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1208 PS9, Line 1208: auto nit: use "const auto&". I think that's why the clang-tidy job failed. http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1217 PS9, Line 1217: auto nit: const auto& http://gerrit.cloudera.org:8080/#/c/21020/9/be/src/service/impala-server.cc@1225 PS9, Line 1225: auto nit: const auto& -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Wed, 21 Feb 2024 08:54:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
IMPALA-12793: Create JDBC table without data source
This patch changes syntax of creating JDBC table statement as
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
(col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
)
[COMMENT 'table_comment']
STORED BY JDBC
TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)
Both "STORED BY JDBC" and "STORED AS JDBC" are working.
Testing:
- Added frontend unit tests for new syntax.
- Updated end-to-end unit tests to create JDBC tables without
data source..
- Passed core tests
Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
---
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
A fe/src/main/java/org/apache/impala/util/JsonUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M testdata/bin/create-ext-data-source-table.sql
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M
testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test
M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
M
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/custom_cluster/test_ext_data_sources.py
24 files changed, 383 insertions(+), 364 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/21016/3
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12786: Optimize count(*) for JSON scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21039 ) Change subject: IMPALA-12786: Optimize count(*) for JSON scans .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15260/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97ff097661c3c577aeafeeb1518408ce7a8a255e Gerrit-Change-Number: 21039 Gerrit-PatchSet: 4 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Feb 2024 08:47:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20986 )
Change subject: IMPALA-12782: Show info of the event processing in /events webUI
..
Patch Set 4:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/20986/3/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/20986/3/be/src/catalog/catalog-server.cc@964
PS3, Line 964: if (progress_info.num_filtered_events > 0) {
> There is a possibility of progress_info.num_filtered_events being 0.
Nice catch!
http://gerrit.cloudera.org:8080/#/c/20986/2/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/20986/2/common/thrift/JniCatalog.thrift@1066
PS2, Line 1066: }
> Nice catch! I planned to show the MetastoreEvent at the beginning but decid
In patch set 3, I finally replace this field with 'current_event_batch_size'
and show it in the page.
http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@678
PS3, Line 678: LOG.info("Ignored exception", e);
> nit: What is your opinion about just logging at this InterruptedException i
Removed this.
http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1137
PS3, Line 1137: progressInfo.current_event_batch_size =
currentFilteredEvent_ != null ?
> There is a possibility of currentFilteredEvent_ being null. Recommend a che
Nice catch!
http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1192
PS3, Line 1192: lastSyncedEventTimeSecs_.set(e.getEventTime());
> Recommend resetProgress() here as well.
Nice catch!
http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1194
PS3, Line 1194: return;
> nit: Need not set currentEventIndex_ explicitly to 0 now.
Done
--
To view, visit http://gerrit.cloudera.org:8080/20986
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c
Gerrit-Change-Number: 20986
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Wed, 21 Feb 2024 08:41:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12782: Show info of the event processing in /events webUI
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20986 to look at the new patch set (#4). Change subject: IMPALA-12782: Show info of the event processing in /events webUI .. IMPALA-12782: Show info of the event processing in /events webUI The /events page of catalogd shows the metrics and status of the event-processor. This patch adds more info in this page, including - lag info - current event batch that's being processing See the screenshot attached in the JIRA for how it looks like. Also moves the error message to the top to highlight the error status. Adds a debug flag, event_processing_delay_s, to inject a sleep while processing an event. So the web page can be captured more easily. Tests: - Add e2e test to verify the content of the page. Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c --- M be/src/catalog/catalog-server.cc M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M be/src/util/pretty-printer.h M common/thrift/BackendGflags.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_web_pages.py M www/events.tmpl 11 files changed, 294 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20986/4 -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12801: Increase query log default size and bound its memory.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21020 ) Change subject: IMPALA-12801: Increase query_log_ default size and bound its memory. .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10286/ -- To view, visit http://gerrit.cloudera.org:8080/21020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I107e2c2c7f2b239557be37360e8eecf5479e8602 Gerrit-Change-Number: 21020 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Wed, 21 Feb 2024 08:33:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12786: Optimize count(*) for JSON scans
Zihao Ye has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21039 Change subject: IMPALA-12786: Optimize count(*) for JSON scans .. IMPALA-12786: Optimize count(*) for JSON scans When performing zero slots scans on a JSON table for operations like count(*), we don't require specific data from the JSON, we only need the number of top-level JSON objects. However, the current JSON parser based on rapidjson still decodes and copies specific data from the JSON, even in zero slots scans. Skipping these steps can significantly improve scan performance. This patch introduces a JSON skipper to conduct zero slots scans on JSON data. Essentially, it is a simplified version of a rapidjson parser, removing specific data decoding and copying operations, resulting in faster parsing of the number of JSON objects. The skipper retains the ability to recognize malformed JSON and provide specific error codes same as the rapidjson parser. Nevertheless, as it bypasses specific data parsing, it cannot identify string encoding errors or numeric overflow errors. Despite this, these data errors do not impact the counting of JSON objects, so it is acceptable to ignore them. The TEXT scanner exhibits similar behavior. Additionally, a new query option, disable_optimized_json_count_star, has been added to disable this optimization and revert to the old behavior. In the performance test of TPC-DS with a format of json/none and a scale of 10GB, the performance optimization is shown in the following tables: +---+---+++-++---++---++-++ | Workload | Query | File Format| Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +---+---+++-++---++---++-++ | TPCDS(10) | TPCDS-Q_COUNT_UNOPTIMIZED | json / none / none | 6.78 | 6.88 | -1.46% | 4.93% | 3.63%| 9 | -1.51% | -0.74 | -0.72 | | TPCDS(10) | TPCDS-Q_COUNT_ZERO_SLOT | json / none / none | 2.42 | 6.75 | I -64.20% | 6.44% | 4.58%| 9 | I -177.75% | -3.36 | -37.55 | | TPCDS(10) | TPCDS-Q_COUNT_OPTIMIZED | json / none / none | 2.42 | 7.03 | I -65.63% | 3.93% | 4.39%| 9 | I -194.13% | -3.36 | -42.82 | +---+---+++-++---++---++-++ (I) Improvement: TPCDS(10) TPCDS-Q_COUNT_ZERO_SLOT [json / none / none] (6.75s -> 2.42s [-64.20%]) +--++-+--+++--+--+++---++---+ | Operator | % of Query | Avg | Base Avg | Delta(Avg) | StdDev(%) | Max | Base Max | Delta(Max) | #Hosts | #Inst | #Rows | Est #Rows | +--++-+--+++--+--+++---++---+ | 01:AGGREGATE | 2.58% | 54.85ms | 58.88ms | -6.85% | * 14.43% * | 115.82ms | 133.11ms | -12.99%| 3 | 3 | 3 | 1 | | 00:SCAN HDFS | 97.41% | 2.07s | 6.07s| -65.84%| 5.87%| 2.43s| 6.95s| -65.01%| 3 | 3 | 28.80M | 143.83M | +--++-+--+++--+--+++---++---+ (I) Improvement: TPCDS(10) TPCDS-Q_COUNT_OPTIMIZED [json / none / none] (7.03s -> 2.42s [-65.63%]) +--++---+--++---+---+--+++---++---+ | Operator | % of Query | Avg | Base Avg | Delta(Avg) | StdDev(%) | Max | Base Max | Delta(Max) | #Hosts | #Inst | #Rows | Est #Rows | +--++---+--++---+---+--+++---++---+ | 00:SCAN HDFS | 99.35% | 2.07s | 6.49s| -68.15%| 4.83% | 2.37s | 7.49s| -68.32%| 3 | 3 | 28.80M | 143.83M | +--++---+--++---+---+--+++---++---+ Testing: - Added new test cases in TestQueriesJsonTables to verify that query results are consistent before and after optimization. - Passed existing JSON scanning-related tests. Change-Id: I97ff097661c3c577aeafeeb1518408ce7a8a255e --- M be/src/exec/json/hdfs-json-scanner.cc M be/src/exec/json/json-parser-test.cc M be/src/exec/json/json-parser.cc M be/src/ex
