[Impala-ASF-CR] IMPALA-11095: Fix Impala-shell strict hs2 mode inserts
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18176 ) Change subject: IMPALA-11095: Fix Impala-shell strict_hs2 mode inserts .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe837c99e54d68d1e27b97f0025e17faf0a2cb9f Gerrit-Change-Number: 18176 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 04 Feb 2022 07:42:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11095: Fix Impala-shell strict hs2 mode inserts
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18176 ) Change subject: IMPALA-11095: Fix Impala-shell strict_hs2 mode inserts .. IMPALA-11095: Fix Impala-shell strict_hs2 mode inserts The insert command was broken for impala-shell in the strict_hs2 mode. The return parameter for close_dml should return two parameters. The parameters returned by close_dml are rows returned and error rows. These are not supported by strict hs2 mode since the close does not return the TDmlResult structure. So the message to the end user also had to be changed. Change-Id: Ibe837c99e54d68d1e27b97f0025e17faf0a2cb9f Reviewed-on: http://gerrit.cloudera.org:8080/18176 Tested-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer --- M shell/impala_client.py M shell/impala_shell.py M tests/shell/test_shell_commandline.py 3 files changed, 14 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Verified Csaba Ringhofer: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibe837c99e54d68d1e27b97f0025e17faf0a2cb9f Gerrit-Change-Number: 18176 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 14 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 04 Feb 2022 07:28:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7795/ -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 04 Feb 2022 06:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11097: Call wait to finish before fetch results for HS2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18133 ) Change subject: IMPALA-11097: Call wait_to_finish before fetch_results for HS2. .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7794/ -- To view, visit http://gerrit.cloudera.org:8080/18133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 Gerrit-Change-Number: 18133 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 04 Feb 2022 06:29:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18195 ) Change subject: IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7797/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41b0c45d9693cb3433e02737048cb9f50ba59c1 Gerrit-Change-Number: 18195 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 04 Feb 2022 06:29:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18195 ) Change subject: IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails .. Patch Set 1: Code-Review+2 This makes sense to me. -- To view, visit http://gerrit.cloudera.org:8080/18195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41b0c45d9693cb3433e02737048cb9f50ba59c1 Gerrit-Change-Number: 18195 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 04 Feb 2022 02:46:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10095/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 04 Feb 2022 02:08:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 8: Add an observability test to observe two instances of compilation events in run-time profile. -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 04 Feb 2022 01:49:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Qifan Chen has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. IMPALA-10992 Planner changes for estimate peak memory This patch provides replan support for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when relevant meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The needed meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. To facilitate testing, the patch imposes an artificial two executor group setup in FE. This setup is enabled when a new query option 'test_replan' is set to 1 or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. This query option is set to 0 by default. The artificial two executor groups are configured as follows. 1. [regular:<#nodes>, 64MB] 2. [large:<#nodes>, 8PB] Compilation time increases when a query is compiled in several iterations, as shown below for several TPCDs queries in two. For small queries, the increase can be minimized if the smallest threshold among all executor group sets is set to catch them. For example, for the queries listed below, the smallest threshold can be set to 320MB to catch query q15 and q21. Compilation time (ms) Queries Estimated Memory 2-iterations 1-iteration q1 408MB 18.32 13.01 q11 1.37GB 186.17 86.28 q10a 519MB 108.27 53.58 q13339MB 118.03 82.43 q14a 3.56GB 628.27307.24 q14b 2.20GB 518.79239.05 q15314MB 13.12 4.51 q21275MB 11.04 6.34 q23a 1.34GB 458.7227.62 q23b 1.50GB 471.29224.75 q42.60GB 206.34 98.64 q67 5.16GB 691.45336.31 Testing: 1. Almost all FE and BE tests are now run in the artificial two executor setup except a few where a specific cluster configuration is desirable; 2. Ran core tests successfully; 3. Added a new observability test. Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M tests/common/test_dimensions.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 21 files changed, 517 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/18178/8 -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/18178/8/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/18178/8/tests/query_test/test_observability.py@772 PS8, Line 772: i flake8: F841 local variable 'impalad' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 04 Feb 2022 01:48:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7796/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 14 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 04 Feb 2022 01:08:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11095: Fix Impala-shell strict hs2 mode inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18176 ) Change subject: IMPALA-11095: Fix Impala-shell strict_hs2 mode inserts .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe837c99e54d68d1e27b97f0025e17faf0a2cb9f Gerrit-Change-Number: 18176 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 04 Feb 2022 01:08:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 04 Feb 2022 00:08:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7795/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 04 Feb 2022 00:08:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 1: The test failure looks unrelated, will retry. -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 04 Feb 2022 00:04:17 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: WIP IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10094/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 04 Feb 2022 00:02:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11097: Call wait to finish before fetch results for HS2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18133 ) Change subject: IMPALA-11097: Call wait_to_finish before fetch_results for HS2. .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7794/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18133 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ab4b90ba2e1a439119d37fe9fb9c55eeeb53ba0 Gerrit-Change-Number: 18133 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Feb 2022 23:58:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7792/ -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Feb 2022 23:58:37 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10898: Add runtime IN-list filters for ORC tables
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18141 to look at the new patch set (#4). Change subject: WIP IMPALA-10898: Add runtime IN-list filters for ORC tables .. WIP IMPALA-10898: Add runtime IN-list filters for ORC tables Currently Impala has two kinds of runtime filters: bloom filter and min-max filter. Unfortunately they can't leverage the bloom filters in ORC files. Because only EQUALS and IN-list predicates can leverage them to skip unrelated ORC RowGroups, and we can't convert runtime bloom filters or min-max filters into such predicates. This patch adds runtime IN-list filters for small build side (e.g. #rows <= 1024) of a broadcast join. Currently the IN-list filters will only apply to ORC tables and be pushed down to the ORC reader(i.e. ORC lib). Evaluating runtime IN-list filters is much slower than evaluating runtime bloom filters due to the current simple implementation (i.e. std::unorder_set). So we disable it at row level. Example query that will benefit from this patch: use tpch_orc_def; select count(*) from lineitem_bf join ( select * from partsupp, part where ps_partkey = p_partkey and p_size = 15 and p_type like '%BRASS' and ps_availqty < 10) v on l_partkey = ps_partkey and l_suppkey = ps_suppkey; The inline-view populates a runtime IN-list filter of 4 items. Note that we need to re-generate the lineitem table with bloom filters enabled (e.g. setting orc.bloom.filter.columns to "l_orderkey,l_partkey,l_suppkey,l_linenumber,l_quantity" in tblproperties), so the pushed down IN-list filter can have a better filter rate. TODO: fix tests due to plan changes. Change-Id: I25080628233799aa0b6be18d5a832f1385414501 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/join-builder.cc M be/src/exec/nested-loop-join-builder.h M be/src/exec/orc-metadata-utils.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/scan-node.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter-ir.cc M be/src/runtime/runtime-filter-test.cc M be/src/runtime/runtime-filter.cc M be/src/runtime/runtime-filter.h M be/src/runtime/runtime-filter.inline.h M be/src/service/data-stream-service.cc M be/src/service/query-options-test.cc M be/src/util/CMakeLists.txt A be/src/util/in-list-filter-ir.cc A be/src/util/in-list-filter.cc A be/src/util/in-list-filter.h M common/protobuf/data_stream_service.proto M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M tests/query_test/test_runtime_filters.py 31 files changed, 748 insertions(+), 122 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/18141/4 -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata If we insert data into an acid partitioned table from Hive, the generated events will be like open_txn -> alter_partition -> commit_txn. Previously we assumed the partition object with the alter_partition event has write id < current write id. However, that is not a valid assumption, the partition object is actually the write id allocated in this transaction. That means in commit_txn event, we will have a partition with write id equals to the write id of cached partition. So we need to modify the '<' condition to '<='. Tests: After IMPALA-10923, we now refresh file metadata while processing commit events. Therefore, we can add back the test disabled in IMPALA-9057. Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Reviewed-on: http://gerrit.cloudera.org:8080/18175 Tested-by: Impala Public Jenkins Reviewed-by: Sourabh Goyal Reviewed-by: Joe McDonnell --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M tests/metadata/test_event_processing.py 2 files changed, 1 insertion(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified Sourabh Goyal: Looks good to me, but someone else must approve Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 21:21:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39 PS3, Line 39: > Yes, the test will fail intermittently without fine-grained table refreshin Thanks for the clarification. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 20:39:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11106: Make Impala compatible with Iceberg 0.13
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18196 ) Change subject: IMPALA-11106: Make Impala compatible with Iceberg 0.13 .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18196/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/18196/1/fe/src/main/java/org/apache/impala/service/Frontend.java@a53 PS1, Line 53: Can we remove the import? -- To view, visit http://gerrit.cloudera.org:8080/18196 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a95483ea3ce15af9a344f9da3b7bfb7dea8a10a Gerrit-Change-Number: 18196 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 03 Feb 2022 19:55:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10093/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 03 Feb 2022 19:13:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6636: Use async IO in ORC scanner
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/15370 )
Change subject: IMPALA-6636: Use async IO in ORC scanner
..
Patch Set 27:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15370/25/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/15370/25/be/src/exec/hdfs-orc-scanner.cc@1375
PS25, Line 1375: stitute("HdfsOrc
> Thanks for the explanation!
Agree. Filed IMPALA-11107 for this.
--
To view, visit http://gerrit.cloudera.org:8080/15370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I348ad9e55f0cae7dff0d74d941b026dcbf5e4074
Gerrit-Change-Number: 15370
Gerrit-PatchSet: 27
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Thu, 03 Feb 2022 19:13:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11106: Make Impala compatible with Iceberg 0.13
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18196 ) Change subject: IMPALA-11106: Make Impala compatible with Iceberg 0.13 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10092/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18196 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a95483ea3ce15af9a344f9da3b7bfb7dea8a10a Gerrit-Change-Number: 18196 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 03 Feb 2022 18:59:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 7: Also include the measurement of compilation time in commit message. -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 03 Feb 2022 18:50:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 7: Address core test failures. The number of nodes in testing mode will be from ExecutorMembershipSnapshot.getCluster()numExecutors(). -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 03 Feb 2022 18:49:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Qifan Chen has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. IMPALA-10992 Planner changes for estimate peak memory This patch provides replan support for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when relevant meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The needed meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. To facilitate testing, the patch imposes an artificial two executor group setup in FE. This setup is enabled when a new query option 'test_replan' is set to 1 or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. This query option is set to 0 by default. The artificial two executor groups are configured as follows. 1. [regular:<#nodes>, 64MB] 2. [large:<#nodes>, 8PB] Compilation time will increase when a query is compiled in several iterations, as shown below for several TPCDs queries. For small queries, the increase can be minimized if the smallest threshold among all executor group sets is set to catch them. For example, for the queries listed below, the smallest threshold is set to 320MB to catch query q15 and q21. Compilation time (ms) Queries Estimated Memory 2-iteration 1-iteration q1408MB 18.32 13.01 q11 1.37GB 186.17 86.28 q10a 519MB 108.27 53.58 q13 339MB 118.03 82.43 q14a 3.56GB 628.27307.24 q14b 2.20GB 518.79239.05 q15 314MB 13.12 4.51 q21 275MB 11.04 6.34 q23a 1.34GB 458.7227.62 q23b 1.50GB 471.29224.75 q4 2.60GB 206.34 98.64 q67 5.16GB 691.45336.31 Testing: 1. Almost all FE and BE tests are now run in the artificial two executor setup except a few where a specific cluster configuration is desirable; 2. Ran core tests. Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M tests/common/test_dimensions.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 21 files changed, 504 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/18178/7 -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 7 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zh
[Impala-ASF-CR] IMPALA-11106: Make Impala compatible with Iceberg 0.13
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18196 Change subject: IMPALA-11106: Make Impala compatible with Iceberg 0.13 .. IMPALA-11106: Make Impala compatible with Iceberg 0.13 Currently we are using deprecated methods from Iceberg API which won't be in 0.13. We need to switch to the newer methods: * PartitionSpecVisitor.visit * HiveCatalog constructor We are also using SnapshotUtil from the Iceberg Core package. One of its methods has changed semantically. We shouldn't depend on packages outside of Iceberg API as they are treated as internal modules. Testing: * built Impala with Iceberg 0.13.0 and executed Iceberg-related tests Change-Id: I7a95483ea3ce15af9a344f9da3b7bfb7dea8a10a --- M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java 3 files changed, 22 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/18196/1 -- To view, visit http://gerrit.cloudera.org:8080/18196 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7a95483ea3ce15af9a344f9da3b7bfb7dea8a10a Gerrit-Change-Number: 18196 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11095: Fix Impala-shell strict hs2 mode inserts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18176 ) Change subject: IMPALA-11095: Fix Impala-shell strict_hs2 mode inserts .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7793/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe837c99e54d68d1e27b97f0025e17faf0a2cb9f Gerrit-Change-Number: 18176 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Feb 2022 18:36:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39 PS3, Line 39: > @Yu-Wen: Please confirm the following: Yes, the test will fail intermittently without fine-grained table refreshing. The issue was that we previously refresh file metadata at alter partition event but while alter partition event was processed the transaction might not be committed yet. If it is committed, we could get new file metadata. Otherwise, we would still see stale file metadata. After my patch, we can now refresh file metadata at commit event. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 18:06:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39 PS3, Line 39: @Yu-Wen: Please confirm the following: 1. Are we enabling the test because the flakiness got fixed? 2. Does this test fail if we enable it without the fix in this patch? -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 17:48:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18195 ) Change subject: IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10091/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41b0c45d9693cb3433e02737048cb9f50ba59c1 Gerrit-Change-Number: 18195 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Feb 2022 17:22:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18195 Change subject: IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails .. IMPALA-11105: Impala crashes in PhjBuilder::Close() when Prepare() fails In PhjBuilder::Close() we invoke 'ht_ctx_->StatsCountersAdd(ht_stats_profile_.get())' when 'ht_ctx_' is not null. But in Prepare we create 'ht_ctx_' first, then after a couple operations which might fail we create 'ht_stats_profile_'. This means if an operation fails in Prepare(), between the creation of 'ht_ctx_' and 'ht_stast_profile_', then later we'll get a SEGFAULT in Close(). This patch restructures the code in PhjBuilder::Prepare(), so at first it creates the counters and profile, then it creates 'ht_ctx_', similarly to what we do in grouping-aggregator.cc. It also modifies HashTableCtx::StatsCountersAdd(), so in release mode it is a no-op if 'profile' is null. Testing: * added a debug action that fails PhjBuilder::Prepare() after the creation of 'ht_ctx_' Change-Id: Id41b0c45d9693cb3433e02737048cb9f50ba59c1 --- M be/src/exec/hash-table.cc M be/src/exec/partitioned-hash-join-builder.cc M testdata/workloads/functional-query/queries/QueryTest/joins.test 3 files changed, 36 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/18195/1 -- To view, visit http://gerrit.cloudera.org:8080/18195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id41b0c45d9693cb3433e02737048cb9f50ba59c1 Gerrit-Change-Number: 18195 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7792/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Feb 2022 16:41:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10046: Switch backend to compile with DWARF 4 debug info
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18194 ) Change subject: IMPALA-10046: Switch backend to compile with DWARF 4 debug info .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6e75845ab137d0a7674289e4b331f682eee5b2 Gerrit-Change-Number: 18194 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Feb 2022 12:42:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10997: Refactor Java Hive UDF code.
Csaba Ringhofer has removed a vote on this change. Change subject: IMPALA-10997: Refactor Java Hive UDF code. .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Idc9572e15fbed1876412159b993fb4d37174 Gerrit-Change-Number: 18020 Gerrit-PatchSet: 18 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-10997: Refactor Java Hive UDF code.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18020 ) Change subject: IMPALA-10997: Refactor Java Hive UDF code. .. IMPALA-10997: Refactor Java Hive UDF code. In its current form, Impala supports Java UDFs that are derived from the UDF.class. The UDF.class is legacy code and Hive now supports implementation off of the GenericUDF.class. This rewrite will allow for easier extension to support GenericUDFs. Among added classes: UdfExecutor: The entry point class which is directly accessed by the backend. This is a wrapper class to the UDF class that will handle the evaluation of rows. HiveUdfExecutor: Abstract base class that contains code that is common to the legacy UDF.class and the GenericUDF.class when it is eventually created. HiveUdfExecutorLegacy: Implementation of the code that is UDF.class specific. HiveUdfLoader: Class responsible for using reflection to instantiate the UDF class HiveJavaFunction: Interface for retrieving objects pertaining to the UDF function class. HiveLegacyJavaFunction: Class representing the metadata for the legacy UDF class. Also added some functionality which captures the error when a user attempts to create a function and the function doesn't exist. The unit test checking this is the UDFRound function which no longer exists in hive-exec.jar so it is now in a load-java-udfs-fail.test test file. Change-Id: Idc9572e15fbed1876412159b993fb4d37174 Reviewed-on: http://gerrit.cloudera.org:8080/18020 Reviewed-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer Tested-by: Csaba Ringhofer --- M fe/src/main/java/org/apache/impala/analysis/CreateUdfStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java A fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java A fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactory.java A fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactoryImpl.java A fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java A fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutor.java A fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutorLegacy.java A fe/src/main/java/org/apache/impala/hive/executor/HiveUdfLoader.java A fe/src/main/java/org/apache/impala/hive/executor/JavaUdfDataType.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/FunctionUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A fe/src/test/java/org/apache/impala/hive/executor/HiveLegacyJavaFunctionTest.java A fe/src/test/java/org/apache/impala/hive/executor/TestHiveJavaFunctionFactory.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java A testdata/workloads/functional-query/queries/QueryTest/load-java-udfs-fail.test M testdata/workloads/functional-query/queries/QueryTest/load-java-udfs.test M tests/query_test/test_udfs.py 24 files changed, 1,652 insertions(+), 816 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Csaba Ringhofer: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idc9572e15fbed1876412159b993fb4d37174 Gerrit-Change-Number: 18020 Gerrit-PatchSet: 19 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-10997: Refactor Java Hive UDF code.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18020 ) Change subject: IMPALA-10997: Refactor Java Hive UDF code. .. Patch Set 18: Verified+1 Code-Review+2 the issue hit was a known flaky test: IMPALA-10927 merging this manually -- To view, visit http://gerrit.cloudera.org:8080/18020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9572e15fbed1876412159b993fb4d37174 Gerrit-Change-Number: 18020 Gerrit-PatchSet: 18 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Thu, 03 Feb 2022 09:13:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6636: Use async IO in ORC scanner
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/15370 )
Change subject: IMPALA-6636: Use async IO in ORC scanner
..
Patch Set 27: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15370/25/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/15370/25/be/src/exec/hdfs-orc-scanner.cc@1375
PS25, Line 1375: stitute("HdfsOrc
> You are correct. Until now, we were reading the last 100KB, but didn't actu
Thanks for the explanation!
I would prefer to reduce the initial range size to 16KB (it is ok to move this
to another patch).
It should be easy to do this by passing a size to
HdfsScanner::IssueFooterRanges instead of using constant:
https://github.com/apache/impala/blob/57982efc21746f6994c11b623fc3cdd1dbbac8a2/be/src/exec/hdfs-scanner.cc#L832
We don't just read something and never use it, but also waste the data cache:
https://github.com/apache/impala/blob/master/be/src/runtime/io/data-cache.h#L73
--
To view, visit http://gerrit.cloudera.org:8080/15370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I348ad9e55f0cae7dff0d74d941b026dcbf5e4074
Gerrit-Change-Number: 15370
Gerrit-PatchSet: 27
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Thu, 03 Feb 2022 08:22:19 +
Gerrit-HasComments: Yes
