[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 26: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 26 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 24 Apr 2024 05:55:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#16). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
TPCDS tests for JDBC tables run only for release/exhaustive builds.
TPCH tests for JDBC tables run for core and exhaustive builds, except
Dockerized builds.
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core tests.
- Passed query_test/test_tpcds_queries.py in release/exhaustive build.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/common/skip.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
23 files changed, 1,914 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/16
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 16
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous
[Impala-ASF-CR] IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21350 ) Change subject: IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16001/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fb357016d72a363ee5016f7881b0f6b0426aff5 Gerrit-Change-Number: 21350 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2024 01:47:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details
Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21350 Change subject: IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details .. IMPALA-13031: Enhancing logging for spilling configuration with local buffer directory details The patch adds logging for local buffer directory when using remote scratch space. The printed log would be like "Using local buffer directory for scratch space /tmp/test/impala-scratch on disk 8 limit: 500.00 MB, priority: 2147483647". Tests: Manally tests the logging working as described. Change-Id: I8fb357016d72a363ee5016f7881b0f6b0426aff5 --- M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr.cc 2 files changed, 14 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/21350/1 -- To view, visit http://gerrit.cloudera.org:8080/21350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8fb357016d72a363ee5016f7881b0f6b0426aff5 Gerrit-Change-Number: 21350 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 26: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10577/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 26 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 24 Apr 2024 00:52:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 26: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10576/ -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 26 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 24 Apr 2024 00:13:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16000/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 15 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 23 Apr 2024 23:10:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#15). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- TPCDS tests run only for release/exhaustive builds.
TPCH tests run for core builds.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/common/skip.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
23 files changed, 1,914 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/15
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 15
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 26: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15999/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 26 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 23 Apr 2024 19:42:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Hello Aman Sinha, Quanlong Huang, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21109 to look at the new patch set (#26). Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. IMPALA-12872: Use Calcite for optimization - part 1: simple queries This is the first commit to use the Calcite library to parse, analyze, and optimize queries. The hook for the planner is through an override of the JniFrontend. The CalciteJniFrontend class is the driver that walks through each of the Calcite steps which are as follows: CalciteQueryParser: Takes the string query and outputs an AST in the form of Calcite's SqlNode object. CalciteMetadataHandler: Iterate through the SqlNode from the previous step and make sure all essential table metadata is retrieved from catalogd. CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer. CalciteRelNodeConverter: Change the AST into a logical plan. In this first commit, the only logical nodes used are LogicalTableScan and LogicalProject. The LogicalTableScan will serve as the node that reads from an Hdfs Table and the LogicalProject will only project out the used columns in the query. In later versions, the LogicalProject will also handle function changes. CalciteOptimizer: This step is to optimize the query. In this cut, it will be a nop, but in later versions, it will perform logical optimizations via Calcite's rule mechanism. CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into Impala's PlanNode physical tree ExecRequestCreator: Implement the existing Impala steps that turn a Single Node Plan into a Distributed Plan. It will also create the TExecRequest object needed by the runtime server. Only some very basic queries will work with this commit. These include: select * from tbl <-- only needs the LogicalTableScan select c1 from tbl <-- Also uses the LogicalProject In the CalciteJniFrontend, there is some basic checks to make sure only select statements will get processed. Any non-query statement will revert back to the current Impala planner. In this iteration, any queries besides the minimal ones listed above will result in a caught exception which will then be run through the current Impala planner. The tests that do work can be found in calcite.test and run through the custom cluster test test_experimental_planner.py This iteration should support all types with the exception of complex types. Calcite does not have a STRING type, so the string type is represented as VARCHAR(MAXINT) similar to how Hive represents their STRING type. The ImpalaTypeConverter file is used to convert the Impala Type object to corresponding Calcite objects. Authorization is not yet working with this current commit. A Jira has been filed (IMPALA-13011) to deal with this. Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 --- M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/calcite-planner/pom.xml A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeWithExprs.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaCalciteCatalogReader.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 26: (1 comment) http://gerrit.cloudera.org:8080/#/c/21109/26/java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java: http://gerrit.cloudera.org:8080/#/c/21109/26/java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java@26 PS26, Line 26: * https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/validate/SqlConformance.html line too long (98 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 26 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 23 Apr 2024 19:19:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 26: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10576/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 26 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 23 Apr 2024 19:18:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 25: (1 comment) http://gerrit.cloudera.org:8080/#/c/21109/25/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/21109/25/fe/src/main/java/org/apache/impala/service/Frontend.java@2144 PS25, Line 2144: addPlannerToProfile(PLANNER); When I comment this out, the custom_cluster/test_query_log.py and custom_cluster/test_query_live.py tests pass. I'll dig a bit, but my guess is that the new line in the profile interacts with the query history table. -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 25 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 23 Apr 2024 17:54:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15998/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 23 Apr 2024 16:53:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 6: Patch set 6 tweak the assertion a bit. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 23 Apr 2024 16:30:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Hello Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21340
to look at the new patch set (#6).
Change subject: IMPALA-13024: Ignore slots if using default pool and empty group
..
IMPALA-13024: Ignore slots if using default pool and empty group
Slot based admission should not be enabled when using default pool.
There is a bug where coordinator-only query still does slot based
admission because executor group name set to
ClusterMembershipMgr::EMPTY_GROUP_NAME ("empty group (using coordinator
only)"). This patch add check to recognize coordinator-only query at
default pool and skip it from slot checking.
Testing:
- Add BE test AdmissionControllerTest.CanAdmitRequestSlotsDefault.
- In test_executor_groups.py, split test_coordinator_concurrency to
test_coordinator_concurrency_default and
test_coordinator_concurrency_two_exec_group_cluster to show the
behavior change.
- Pass core tests in ASAN build.
Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
---
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/request-pool-service.h
M tests/common/impala_connection.py
M tests/custom_cluster/test_executor_groups.py
9 files changed, 126 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/21340/6
--
To view, visit http://gerrit.cloudera.org:8080/21340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
Gerrit-Change-Number: 21340
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR](asf-site) Add documentation, update links for 4.4.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21311 ) Change subject: Add documentation, update links for 4.4.0 .. Patch Set 3: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/763/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/21311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Ibb93f7ba80b7a065ea83660fc75be9b065138ad9 Gerrit-Change-Number: 21311 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Apr 2024 16:11:12 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add documentation, update links for 4.4.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21311 ) Change subject: Add documentation, update links for 4.4.0 .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/763/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/21311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Ibb93f7ba80b7a065ea83660fc75be9b065138ad9 Gerrit-Change-Number: 21311 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Apr 2024 16:07:53 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add documentation, update links for 4.4.0
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21311 to look at the new patch set (#3). Change subject: Add documentation, update links for 4.4.0 .. Add documentation, update links for 4.4.0 Change-Id: Ibb93f7ba80b7a065ea83660fc75be9b065138ad9 --- M docs/build/asf-site-html/index.html M docs/build/asf-site-html/shared/ImpalaVariables.html M docs/build/asf-site-html/shared/impala_common.html M docs/build/asf-site-html/topics/impala_abort_on_error.html M docs/build/asf-site-html/topics/impala_adls.html M docs/build/asf-site-html/topics/impala_admin.html M docs/build/asf-site-html/topics/impala_admission.html M docs/build/asf-site-html/topics/impala_admission_config.html M docs/build/asf-site-html/topics/impala_aggregate_functions.html M docs/build/asf-site-html/topics/impala_aliases.html M docs/build/asf-site-html/topics/impala_allow_erasure_coded_files.html M docs/build/asf-site-html/topics/impala_allow_unsupported_formats.html M docs/build/asf-site-html/topics/impala_alter_database.html M docs/build/asf-site-html/topics/impala_alter_table.html M docs/build/asf-site-html/topics/impala_alter_view.html M docs/build/asf-site-html/topics/impala_analytic_functions.html M docs/build/asf-site-html/topics/impala_appx_count_distinct.html M docs/build/asf-site-html/topics/impala_appx_median.html M docs/build/asf-site-html/topics/impala_array.html M docs/build/asf-site-html/topics/impala_auditing.html M docs/build/asf-site-html/topics/impala_authentication.html M docs/build/asf-site-html/topics/impala_authorization.html M docs/build/asf-site-html/topics/impala_avg.html M docs/build/asf-site-html/topics/impala_avro.html M docs/build/asf-site-html/topics/impala_batch_size.html M docs/build/asf-site-html/topics/impala_bigint.html M docs/build/asf-site-html/topics/impala_bit_functions.html M docs/build/asf-site-html/topics/impala_boolean.html M docs/build/asf-site-html/topics/impala_breakpad.html M docs/build/asf-site-html/topics/impala_broadcast_bytes_limit.html M docs/build/asf-site-html/topics/impala_buffer_pool_limit.html M docs/build/asf-site-html/topics/impala_char.html M docs/build/asf-site-html/topics/impala_client.html M docs/build/asf-site-html/topics/impala_comment.html M docs/build/asf-site-html/topics/impala_comments.html M docs/build/asf-site-html/topics/impala_complex_types.html M docs/build/asf-site-html/topics/impala_components.html M docs/build/asf-site-html/topics/impala_compression_codec.html M docs/build/asf-site-html/topics/impala_compute_stats.html M docs/build/asf-site-html/topics/impala_compute_stats_min_sample_size.html M docs/build/asf-site-html/topics/impala_concepts.html M docs/build/asf-site-html/topics/impala_conditional_functions.html M docs/build/asf-site-html/topics/impala_config.html M docs/build/asf-site-html/topics/impala_config_options.html M docs/build/asf-site-html/topics/impala_config_performance.html M docs/build/asf-site-html/topics/impala_connecting.html M docs/build/asf-site-html/topics/impala_conversion_functions.html M docs/build/asf-site-html/topics/impala_count.html M docs/build/asf-site-html/topics/impala_create_database.html M docs/build/asf-site-html/topics/impala_create_function.html M docs/build/asf-site-html/topics/impala_create_role.html M docs/build/asf-site-html/topics/impala_create_table.html M docs/build/asf-site-html/topics/impala_create_view.html M docs/build/asf-site-html/topics/impala_custom_timezones.html M docs/build/asf-site-html/topics/impala_data_cache.html M docs/build/asf-site-html/topics/impala_databases.html M docs/build/asf-site-html/topics/impala_datatypes.html M docs/build/asf-site-html/topics/impala_date.html M docs/build/asf-site-html/topics/impala_datetime_functions.html M docs/build/asf-site-html/topics/impala_ddl.html M docs/build/asf-site-html/topics/impala_debug_action.html M docs/build/asf-site-html/topics/impala_decimal.html M docs/build/asf-site-html/topics/impala_decimal_v2.html M docs/build/asf-site-html/topics/impala_dedicated_coordinator.html M docs/build/asf-site-html/topics/impala_default_file_format.html M docs/build/asf-site-html/topics/impala_default_hints_insert_statement.html M docs/build/asf-site-html/topics/impala_default_join_distribution_mode.html M docs/build/asf-site-html/topics/impala_default_spillable_buffer_size.html M docs/build/asf-site-html/topics/impala_default_transactional_type.html M docs/build/asf-site-html/topics/impala_delegation.html M docs/build/asf-site-html/topics/impala_delete.html M docs/build/asf-site-html/topics/impala_delete_stats_in_truncate.html M docs/build/asf-site-html/topics/impala_describe.html M docs/build/asf-site-html/topics/impala_development.html M docs/build/asf-site-html/topics/impala_disable_codegen.html M docs/build/asf-site-html/topics/impala_disable_codegen_rows_threshold.html M
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10575/ -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 9 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 23 Apr 2024 15:46:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13029: Tests for multi format equality deletes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21348 ) Change subject: IMPALA-13029: Tests for multi format equality deletes .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15997/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f0ebf7f4d401877741eb3e1c990f1318ac2b4ba Gerrit-Change-Number: 21348 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Apr 2024 15:46:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 23 Apr 2024 15:44:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13029: Tests for multi format equality deletes
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/21348 )
Change subject: IMPALA-13029: Tests for multi format equality deletes
..
Patch Set 1:
(1 comment)
Thanks for adding more tests!
http://gerrit.cloudera.org:8080/#/c/21348/1/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/21348/1/testdata/data/README@1193
PS1, Line 1193:set tblproperties ('write.format.default'='avro');
Would it be possible to do schema evolution + Avro delete files? I.e. using
different delete columns in the Avro eq delete files, to make sure we use the
correct Avro schema in the delete scans.
--
To view, visit http://gerrit.cloudera.org:8080/21348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7f0ebf7f4d401877741eb3e1c990f1318ac2b4ba
Gerrit-Change-Number: 21348
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 23 Apr 2024 15:37:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13029: Tests for multi format equality deletes
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21348 Change subject: IMPALA-13029: Tests for multi format equality deletes .. IMPALA-13029: Tests for multi format equality deletes So far we only had test coverage for Parquet equality deletes. This patch adds new tests where we have equality deletes also in ORC and AVRO. Change-Id: I7f0ebf7f4d401877741eb3e1c990f1318ac2b4ba --- M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/data/0-10-937fe984-34d8-4351-a419-842ad9d30758-2.orc A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/data/0-11-1bd90c11-bab5-4f53-9cd9-7bf85adaa97a-2.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/data/0-9-77e491ad-7b25-4b81-b10a-53b383ae0355-2.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/data/61480fd29dfdfefb-48a79ca5_1414128186_data.0.parq A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/1dd8f48c-c2ef-4239-8fb2-25bcbef7026c-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/454a3281-55e9-4e00-a8a2-fac5c23ff043-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/61ebd3da-da91-443e-9413-2a010f77443b-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/dab4e396-8cc2-4f57-b856-3864822ab5d3-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/snap-1904885991593677469-1-1dd8f48c-c2ef-4239-8fb2-25bcbef7026c.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/snap-1935861967137943703-1-454a3281-55e9-4e00-a8a2-fac5c23ff043.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/snap-4400093814370842303-1-61ebd3da-da91-443e-9413-2a010f77443b.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/snap-8244791200683984727-1-dab4e396-8cc2-4f57-b856-3864822ab5d3.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v3.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v4.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v5.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v6.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/v7.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_multi_format_equality_deletes/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-equality-deletes.test M tests/query_test/test_iceberg.py 25 files changed, 892 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/21348/1 -- To view, visit http://gerrit.cloudera.org:8080/21348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7f0ebf7f4d401877741eb3e1c990f1318ac2b4ba Gerrit-Change-Number: 21348 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21340/4/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/21340/4/tests/custom_cluster/test_executor_groups.py@399 PS4, Line 399: @pytest.mark.execute_serially > Isn't this a trivial query? By default enable_trivial_query_for_admission Done. Replaced the test query with trivial scan + debug action. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 23 Apr 2024 15:21:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Hello Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21340
to look at the new patch set (#5).
Change subject: IMPALA-13024: Ignore slots if using default pool and empty group
..
IMPALA-13024: Ignore slots if using default pool and empty group
Slot based admission should not be enabled when using default pool.
There is a bug where coordinator-only query still does slot based
admission because executor group name set to
ClusterMembershipMgr::EMPTY_GROUP_NAME ("empty group (using coordinator
only)"). This patch add check to recognize coordinator-only query at
default pool and skip it from slot checking.
Testing:
- Add BE test AdmissionControllerTest.CanAdmitRequestSlotsDefault.
- In test_executor_groups.py, split test_coordinator_concurrency to
test_coordinator_concurrency_default and
test_coordinator_concurrency_two_exec_group_cluster to show the
behavior change.
- Pass core tests in ASAN build.
Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
---
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/request-pool-service.h
M tests/common/impala_connection.py
M tests/custom_cluster/test_executor_groups.py
9 files changed, 113 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/21340/5
--
To view, visit http://gerrit.cloudera.org:8080/21340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
Gerrit-Change-Number: 21340
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12950: Improve error message in case of out-of-range numeric conversions
Peter Rozsa has posted comments on this change. (
http://gerrit.cloudera.org:8080/21331 )
Change subject: IMPALA-12950: Improve error message in case of out-of-range
numeric conversions
..
Patch Set 3: Code-Review+1
(1 comment)
Thank you, Daniel!
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc
File be/src/exprs/cast-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc@76
PS2, Line 76: } else {
> It's a good point. These are the only types we'd like to cover here, so ori
I think we have a bunch of mediocre options to choose from:
1. Use a default constant that will be printed when we pass an unknown type:
makes the whole type enforcing a bit weaker as its main goal is to provide a
name just for the defined types.
2. Using the static_assert trick with SFINAE yields ill-formed code
3. Using throw clause: ill-formed code
4. Using DCHECK: needs to duplicate the type names, and I'm not sure that the
internals of DCHECK are feasible for constexpr functions
+1: consteval from C++20 :)
In my opinion, the least concerning is the third one, what you added in the
third patchset, in the future, in case of an upgrade to C++20 it could be
replaced with a consteval function.
Any further inputs from other reviewers are welcome, as it's a tough choice.
--
To view, visit http://gerrit.cloudera.org:8080/21331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32
Gerrit-Change-Number: 21331
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
Gerrit-Comment-Date: Tue, 23 Apr 2024 14:55:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12950: Improve error message in case of out-of-range numeric conversions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21331 ) Change subject: IMPALA-12950: Improve error message in case of out-of-range numeric conversions .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15995/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32 Gerrit-Change-Number: 21331 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Tue, 23 Apr 2024 13:27:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12950: Improve error message in case of out-of-range numeric conversions
Daniel Becker has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21331 ) Change subject: IMPALA-12950: Improve error message in case of out-of-range numeric conversions .. IMPALA-12950: Improve error message in case of out-of-range numeric conversions IMPALA-12035 introduced checks for numeric conversions that are unsafe and can fail (if the target type cannot store the value, the behaviour is undefined): - from floating-point types to integer types - from double to float However, it can be difficult to trace which part of the query caused this based on the error message. This change adds the source type, the destination type and the value to be converted to the error message. Unfortunately, at this point in the BE, the original SQL is not available, so we cannot reference that. Testing: - extended existing tests in expr-test.cc. Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32 --- M be/src/exprs/cast-functions-ir.cc M be/src/exprs/expr-test.cc M be/src/udf/udf.h 3 files changed, 85 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21331/3 -- To view, visit http://gerrit.cloudera.org:8080/21331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32 Gerrit-Change-Number: 21331 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-12950: Improve error message in case of out-of-range numeric conversions
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/21331 )
Change subject: IMPALA-12950: Improve error message in case of out-of-range
numeric conversions
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21331/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21331/2//COMMIT_MSG@7
PS2, Line 7: :
> nit: missing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/21331/2//COMMIT_MSG@12
PS2, Line 12: floating-point
> nit: floating-point
Done
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc
File be/src/exprs/cast-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc@76
PS2, Line 76: } else {
> The default case is missing, it could be added as "UNKNOWN TYPE" or somethi
It's a good point. These are the only types we'd like to cover here, so
originally I wanted to add
static_assert(false);
but it doesn't compile. I could write
static_assert(!std::is_same_v), which is always false, however this
link suggests even that may be ill-formed:
https://stackoverflow.com/questions/38304847/constexpr-if-and-static-assert
On the other hand, this latter approach seems to work in practice, but I'm not
sure we should do that if it's not guaranteed. Leaving out the default case
deterministically leads to a warning about no return statement so I did it like
this.
Maybe adding a DCHECK would be best, but I don't like that there doesn't seem
to be a clean and concise way of doing it compile time without repeating the
types (e.g. static_assert that T is one of the types, or some SFINAE magic).
In the new patch set I went with static_assert(!std::is_same_v), it
correctly fires when I try to instantiate the template with a different type,
but I don't know if it's guaranteed or not. What do you think?
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc@182
PS2, Line 182: constexpr const char* FROM_TYPE_NAME =
TypeToName();
> Could you please add test cases to cover each condition?
Extended existing tests and added some new ones in expr-test.cc.
--
To view, visit http://gerrit.cloudera.org:8080/21331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32
Gerrit-Change-Number: 21331
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
Gerrit-Comment-Date: Tue, 23 Apr 2024 13:03:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. IMPALA-13015: Dataload fails due to concurrency issue with test.jceks Move 'hadoop credential' command used for creating test.jceks to testdata/bin/create-load-data.sh. Earlier it was in bin/load-data.py which is called in parallel and was causing failures due to race conditions. Testing: - Ran JniFrontendTest#testGetSecretFromKeyStore after data loading and test ran clean. Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Reviewed-on: http://gerrit.cloudera.org:8080/21346 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/load-data.py M testdata/bin/create-load-data.sh 2 files changed, 10 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 11:09:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10575/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 9 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 23 Apr 2024 10:40:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 9 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 23 Apr 2024 10:40:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13001: Support graceful and force shutdown for impala.sh.
Zihao Ye has posted comments on this change. ( http://gerrit.cloudera.org:8080/21297 ) Change subject: IMPALA-13001: Support graceful and force shutdown for impala.sh. .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21297/1/package/bin/impala.sh File package/bin/impala.sh: http://gerrit.cloudera.org:8080/#/c/21297/1/package/bin/impala.sh@87 PS1, Line 87: counts=20 period=2 > This idea is good and that's what I wanted to do at first, but that two arg Getting those two parameters is indeed tricky. One possible direction might be to try to parse these parameters from the service_flagfile (if configured), or else use the default values for these two parameters (120 and 3600). However, this might make the script overly complicated, and the decision is yours. But at least during a graceful shutdown, making the script timeout match the default values would be more reasonable. I would lean towards matching it with shutdown_grace_period_s and delaying it a bit more appropriately. http://gerrit.cloudera.org:8080/#/c/21297/2/package/bin/impala.sh File package/bin/impala.sh: http://gerrit.cloudera.org:8080/#/c/21297/2/package/bin/impala.sh@261 PS2, Line 261: restart It appears to also support -f/-g. -- To view, visit http://gerrit.cloudera.org:8080/21297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7743234952ba6b12694ecc68a920d59fea0d4ba Gerrit-Change-Number: 21297 Gerrit-PatchSet: 2 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 23 Apr 2024 09:51:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Tue, 23 Apr 2024 09:06:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21301 ) Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly .. IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly If the Iceberg table has Avro delete files (e.g. by setting 'write.delete.format.default'='avro') then Impala won't be able to read the contents of the delete files properly. It is because the avro schema is not set properly for the virtual delete table. Testing: * added e2e tests with position delete files of all kinds Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Reviewed-on: http://gerrit.cloudera.org:8080/21301 Tested-by: Impala Public Jenkins Reviewed-by: Daniel Becker Reviewed-by: Gabor Kaszab --- M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test M tests/query_test/test_iceberg.py 3 files changed, 143 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Daniel Becker: Looks good to me, but someone else must approve Gabor Kaszab: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Gerrit-Change-Number: 21301 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/21301 ) Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21301 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6 Gerrit-Change-Number: 21301 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Apr 2024 08:54:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21326 ) Change subject: IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21326/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21326/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1125 PS5, Line 1125: collected from a new version > If the partition was readded, shouldn't that operation also remove it from > dropped_partitions? I think you mean 'droppedPartitions' of HdfsTable instead of 'dropped_partitions' of THdfsTable which never changes when it's added to the deleteLog. For 'droppedPartitions' of HdfsTable, we haven't done that yet. Currently, it only adds new items in HdfsTable#dropPartition() https://github.com/apache/impala/blob/9b05a205fec397fa1e19ae467b1cc406ca43d948/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1146 We can update it in HdfsTable#addPartitionNoThrow() when a partition is re-added. But that only helps when dropping and re-adding a partition on the same HdfsTable object. That comes to the other question. > How could the catalog collect the new version of the partition before > collecting the deletion of the partition? An example is the following sequence: #1 DropPartition addes the partition to 'droppedPartitions' of HdfsTable #2 InvalidateTable replaces the HdfsTable with an IncompleteTable and adds the THdfsTable object into the deleteLog. The 'dropped_partitions' of this THdfsTable object will have a THdfsPartition object representing this partition. https://github.com/apache/impala/blob/9b05a205fec397fa1e19ae467b1cc406ca43d948/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2363 #3 The table is loaded again so the IncompleteTable is replaced with a new HdfsTable object. #4 AddPartition adds a new HdfsPartition instance (but the same partition name) to the new HdfsTable object. If all these happens in a catalog update cycle, i.e. catalogd collects last round of catalog updates before #1, catalogd will first collect both the table and partition updates at L1013, then collects deletions based on the deleteLog at L1039 and come here. PS5 adds a test case (Test 2) for this: https://gerrit.cloudera.org/c/21326/4..5/tests/custom_cluster/test_partition.py -- To view, visit http://gerrit.cloudera.org:8080/21326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21 Gerrit-Change-Number: 21326 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Apr 2024 07:42:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12950:Improve error message in case of out-of-range numeric conversions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21331 ) Change subject: IMPALA-12950:Improve error message in case of out-of-range numeric conversions .. Patch Set 2: Code-Review+1 lgtm (besides Peter's comments) -- To view, visit http://gerrit.cloudera.org:8080/21331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32 Gerrit-Change-Number: 21331 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Tue, 23 Apr 2024 06:20:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21326 ) Change subject: IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21326/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21326/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1125 PS5, Line 1125: collected from a new version How could the catalog collect the new version of the partition before collecting the deletion of the partition? If the partition was readded, shouldn't that operation also remove it from dropped_partitions? -- To view, visit http://gerrit.cloudera.org:8080/21326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21 Gerrit-Change-Number: 21326 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Apr 2024 06:14:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21340/4/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/21340/4/tests/custom_cluster/test_executor_groups.py@399 PS4, Line 399: QUERY = "select sleep(1000)" Isn't this a trivial query? By default enable_trivial_query_for_admission = true When I reproduced the issue, I used select * from functional.alltypestiny, which has very low cardinality but still has to scan a table. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 23 Apr 2024 06:05:42 + Gerrit-HasComments: Yes
