[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. IMPALA-12896: Avoid JDBC table to be set as transactional table In some deployment environment, JDBC tables are set as transactional tables by default. This causes catalogd failed to load the metadata for JDBC tables. This patch explicitly add table properties with "transactional=false" for JDBC table to avoid the JDBC to be set as transactional table. The operations on JDBC table are processed only on coordinator. The processed rows should be estimated as 0 for DataSourceScanNode by planner so that coordinator-only query plans are generated for simple queries on JDBC tables and queries could be executed without invoking executor nodes. Also adds Preconditions.check to make sure numNodes equals 1 for DataSourceScanNode. Updates FileSystemUtil.copyFileFromUriToLocal() function to write log message for all types of exceptions. Testing: - Fixed planer tests for data source tables. - Ran end-to-end tests of JDBC tables with query option 'exec_single_node_rows_threshold' as default value 100. - Passed core-tests. Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Reviewed-on: http://gerrit.cloudera.org:8080/21141 Reviewed-by: Riza Suminto Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test M tests/custom_cluster/test_ext_data_sources.py M tests/query_test/test_ext_data_sources.py 7 files changed, 42 insertions(+), 13 deletions(-) Approvals: Riza Suminto: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 13 Mar 2024 08:01:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10374/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 13 Mar 2024 03:09:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15495/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 13 Mar 2024 03:05:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21141 )
Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table
..
Patch Set 3: Code-Review+2
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java
File fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java:
http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java@36
PS1, Line 36: // Max number of rows processed across all instances of a plan
node.
: private long maxRowsProcessed_ = 0;
:
: // Max number of rows processed per backend impala daemon for a
plan node.
: private long maxRowsProcessedPerNode_
> fixed
Done
http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java@49
PS1, Line 49: // Operations on DataSourceScanNode are processed on coordinator.
: if (fragment == null) {
: numNodes = ((DataSourceScanNode)caller).getNumNodes();
> fixed as suggested
Done
http://gerrit.cloudera.org:8080/#/c/21141/2/tests/custom_cluster/test_ext_data_sources.py
File tests/custom_cluster/test_ext_data_sources.py:
http://gerrit.cloudera.org:8080/#/c/21141/2/tests/custom_cluster/test_ext_data_sources.py@29
PS2, Line 29:
:
: class TestExtDataSources(CustomClusterTestSuite):
> Fixed as suggested. Thanks
Done
--
To view, visit http://gerrit.cloudera.org:8080/21141
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6
Gerrit-Change-Number: 21141
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 13 Mar 2024 02:55:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@809 PS1, Line 809: Exception > nit: I'd suggest to narrow it down to subclass of Exception that is > actually thrown, but I guess this is fine too to catch all. I will keep the code to catch all exception. http://gerrit.cloudera.org:8080/#/c/21141/2/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21141/2/tests/custom_cluster/test_ext_data_sources.py@29 PS2, Line 29: : : class TestExtDataSources(CustomClusterTestSuite): > You can add_test_dimension to apply exec_single_node_rows_threshold = 100 f Fixed as suggested. Thanks -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 13 Mar 2024 02:42:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. IMPALA-12896: Avoid JDBC table to be set as transactional table In some deployment environment, JDBC tables are set as transactional tables by default. This causes catalogd failed to load the metadata for JDBC tables. This patch explicitly add table properties with "transactional=false" for JDBC table to avoid the JDBC to be set as transactional table. The operations on JDBC table are processed only on coordinator. The processed rows should be estimated as 0 for DataSourceScanNode by planner so that coordinator-only query plans are generated for simple queries on JDBC tables and queries could be executed without invoking executor nodes. Also adds Preconditions.check to make sure numNodes equals 1 for DataSourceScanNode. Updates FileSystemUtil.copyFileFromUriToLocal() function to write log message for all types of exceptions. Testing: - Fixed planer tests for data source tables. - Ran end-to-end tests of JDBC tables with query option 'exec_single_node_rows_threshold' as default value 100. - Passed core-tests. Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 --- M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test M tests/custom_cluster/test_ext_data_sources.py M tests/query_test/test_ext_data_sources.py 7 files changed, 42 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/21141/3 -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15491/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 12 Mar 2024 23:23:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/21141/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@809 PS1, Line 809: Exception > IOException is subclass of Exception. We found the exception was not catche nit: I'd suggest to narrow it down to subclass of Exception that is actually thrown, but I guess this is fine too to catch all. http://gerrit.cloudera.org:8080/#/c/21141/2/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21141/2/tests/custom_cluster/test_ext_data_sources.py@29 PS2, Line 29: : class TestExtDataSources(CustomClusterTestSuite): : """Impala query tests for external data sources.""" You can add_test_dimension to apply exec_single_node_rows_threshold = 100 for all tests here. from tests.common.test_dimensions import create_exec_option_dimension ... @classmethod def add_test_dimensions(cls): super(TestExtDataSources, cls).add_test_dimensions() cls.ImpalaTestMatrix.add_dimension(create_exec_option_dimension( exec_single_node_option=[100])) Similarly for TestImpalaExtJdbcTables. L52 and L254 then can be removed. -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 12 Mar 2024 23:12:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12896: Avoid JDBC table to be set as transactional table
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21141 ) Change subject: IMPALA-12896: Avoid JDBC table to be set as transactional table .. IMPALA-12896: Avoid JDBC table to be set as transactional table In some deployment environment, JDBC tables are set as transactional tables by default. This causes catalogd failed to load the metadata for JDBC tables. This patch explicitly add table properties with "transactional=false" for JDBC table to avoid the JDBC to be set as transactional table. The operations on JDBC table are processed only on coordinator. The processed rows should be estimated as 0 for DataSourceScanNode by planner so that coordinator-only query plans are generated for simple queries on JDBC tables and queries could be executed without invoking executor nodes. Also adds Preconditions.check to make sure numNodes equals 1 for DataSourceScanNode. Updates FileSystemUtil.copyFileFromUriToLocal() function to write log message for all types of exceptions. Testing: - Fixed planer tests for data source tables. - Passed core-tests. Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 --- M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test M tests/custom_cluster/test_ext_data_sources.py M tests/query_test/test_ext_data_sources.py 7 files changed, 33 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/21141/2 -- To view, visit http://gerrit.cloudera.org:8080/21141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I556faeda923a4a11d4bef8c1250c9616f77e6fa6 Gerrit-Change-Number: 21141 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
