[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. IMPALA-9030: Handle translated external Kudu tables In HMS 3.1 there is a default tranformer introduced which checks the client capabilities and transforms a table before creating it. Additionally, it also makes sure that any managed table which is created is transactional. If a user creates a managed table which is not transactional, it automatically converts such table as external and sets certain table properties to mark such transformed tables. This presents a problem for managed Kudu tables in Impala since managed and external tables are handled differently in Kudu. Specifically, if a Kudu table is managed, certain operations like drop table, rename table, alter table are performed on the Kudu side along with updating the catalog. If the Kudu table is external, the Kudu operations are skipped and only catalog side operations are performed. When the user creates a managed Kudu table, user expects that drop table, rename table should be updated by Impala automatically in Kudu as well. But since HMS 3 transforms such managed tables into external, currently Impala does not perform the Kudu side operations breaking the semantics for the user. This patch makes changes to Catalog so that it can detect such transformed external tables and perform Kudu side operations similar to what it was doing for managed Kudu table when talking with previous HMS versions. Note that this change is in preparation of bumping up the CDP build which will be done in a separate change. For the current CDP build number the patch is essentially a no-op. Testing: 1. Bumped up the CDP build number in a private build so that the HMS translation logic is pulled in. Ran all the tests. Without the patch there are many Kudu tests which were failing. After the patch none of the Kudu tests fail. There were additional Ranger tests which failed due to the CDP bump but those were unrelated to this patch and should be fixed as part of a separate change when the CDP build number is bumped up. Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Reviewed-on: http://gerrit.cloudera.org:8080/14397 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 8 files changed, 105 insertions(+), 40 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. IMPALA-9019: fix runtime filter propagation with mt_dop The bug was that filter routing table construction removed filters from the TPlanNode structure for the join when a finstance was not a producer of that filter. The TPlanNode is shared between all instances of a fragment on a backend, so this meant that the filter was removed for all instances on that backend, often meaning that no filters would be produced at all. It was awkward fixing the bug within the framework of the current data structures, where the routing table is keyed by filter_id, so I ended up refactoring the routing table somewhat. This also allowed fixing a TODO about O(n^2) construction of the routing table. Testing: Add regression test that timed out without fix. Perf: Ran a single node TPC-H workload with scale factor 30. No perf change. Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Reviewed-on: http://gerrit.cloudera.org:8080/14511 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M common/thrift/ImpalaInternalService.thrift M testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test 7 files changed, 132 insertions(+), 90 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 22 Oct 2019 04:31:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 04:21:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4850/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 00:44:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 00:30:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. IMPALA-8997: auto fallback to mt_dop=0 Add a temporary --mt_dop_auto_fallback to allow a graceful transition to using mt_dop for workloads. When this flag is set, DML queries and joins that would otherwise fail with an error when run with mt_dop > 0 fall back to running with mt_dop = 0. This means that a user can set mt_dop for their queries and it will only take effect when supported. The behaviour generally does not change when this flag is not set, with a couple of exceptions: * I made mt_dop automatic for compute stats on all file formats * mt_dop is allowed for single node plans with inserts. The quirky validatePlan() logic previously disallowed this but allowed joins in single node plans. The checks added by this patch can be removed safely once mt_dop is supported by default for all queries. This includes some cleanup: * isDmlStmt() was stale and incorrectly implemented. * Various TreeNode methods did not return instances of subclasses of the requested class, which was strange. This fix is required to make 'contains(JoinNode.class)' work correctly. I checked the callsites of the fixed functions and none of them would be affected by this change because they specified a terminal class without any subclasses. I didn't actually use this fix in the end (I had to write a custom tree traversal in hasUnsupportedMtDopJoin()), but figured I would leave the improvement in here. Testing: Add some basic functional tests ensuring that the fallback takes effect. Run basic join and insert tests with this flag enabled. Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Reviewed-on: http://gerrit.cloudera.org:8080/14344 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-auto-fallback.test M tests/custom_cluster/test_mt_dop.py M tests/query_test/test_cancellation.py 16 files changed, 177 insertions(+), 62 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 22 Oct 2019 00:13:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5127/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 22 Oct 2019 00:13:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5126/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 00:00:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 10: Code-Review+2 > Patch Set 10: > > > Patch Set 10: Verified-1 > > > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5123/ > > The failure test TestEventProcessing.test_insert_events seems to be unrelated > flaky test. Agree with that. Codes look good to me. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 22 Oct 2019 00:12:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5120/ -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 22 Oct 2019 00:02:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5 parse name() instead
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/14433 )
Change subject: IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use
krb5_parse_name() instead
..
Patch Set 5:
(7 comments)
This is all looking close. I have some picky comments
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG@10
PS5, Line 10: creating our own.
instead of using custom code.
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG@12
PS5, Line 12: As src/kudu/security/init.cc already have g_krb5_ctx initialized,
The reader of commit messages is developers like you and me.
Will the reader know what 'g_krb5_ctx' is?
I think you want to say something like:
"When kerberos is initialized in Impala's copy of Kudu code, it stores a global
context which is used when accessing the Krb5 library. To use this global
context the code that parses the principal name is moved into the Impala Kudu
code. This new code is then called from the existing ParseKerberosPrincipal
method."
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG@18
PS5, Line 18: Add an authentication-test to verify principal with special
character.
to verify parsing a principal nae containing a special character.
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG@19
PS5, Line 19: Manually tested with bad format principal, throw out error code 2
Is it possible to have an automated test for this?
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG@21
PS5, Line 21:
Do you want to mention running end-to-end tests?
http://gerrit.cloudera.org:8080/#/c/14433/5/be/src/kudu/security/init.h
File be/src/kudu/security/init.h:
http://gerrit.cloudera.org:8080/#/c/14433/5/be/src/kudu/security/init.h@39
PS5, Line 39: // Convert a string representation of a principal name to a
krb5_principal structure.
Maybe: "Parse a kerberos principal name and extract the ervice_name, hostname,
and realm from it."
http://gerrit.cloudera.org:8080/#/c/14433/5/be/src/util/auth-util.cc
File be/src/util/auth-util.cc:
http://gerrit.cloudera.org:8080/#/c/14433/5/be/src/util/auth-util.cc@92
PS5, Line 92: hostname, realm),"bad principal format " + principal);
It is more idiomatic and marginally more efficient to use:
Substitute("bad principal name $0", principal)
--
To view, visit http://gerrit.cloudera.org:8080/14433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24
Gerrit-Change-Number: 14433
Gerrit-PatchSet: 5
Gerrit-Owner: Xiaomeng Zhang
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Xiaomeng Zhang
Gerrit-Comment-Date: Tue, 22 Oct 2019 00:08:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14511 to look at the new patch set (#8). Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. IMPALA-9019: fix runtime filter propagation with mt_dop The bug was that filter routing table construction removed filters from the TPlanNode structure for the join when a finstance was not a producer of that filter. The TPlanNode is shared between all instances of a fragment on a backend, so this meant that the filter was removed for all instances on that backend, often meaning that no filters would be produced at all. It was awkward fixing the bug within the framework of the current data structures, where the routing table is keyed by filter_id, so I ended up refactoring the routing table somewhat. This also allowed fixing a TODO about O(n^2) construction of the routing table. Testing: Add regression test that timed out without fix. Perf: Ran a single node TPC-H workload with scale factor 30. No perf change. Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M common/thrift/ImpalaInternalService.thrift M testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test 7 files changed, 132 insertions(+), 90 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14511/8 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 00:00:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14511 )
Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14511/7/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/14511/7/be/src/runtime/coordinator.cc@127
PS7, Line 127: if (filter_mode_ != TRuntimeFilterMode::OFF) {
Noticed this stale comment so will just delete to avoid confusion.
--
To view, visit http://gerrit.cloudera.org:8080/14511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669
Gerrit-Change-Number: 14511
Gerrit-PatchSet: 8
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 22 Oct 2019 00:00:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/14511/7/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/14511/7/be/src/runtime/coordinator.cc@379 PS7, Line 379: DCHECK(filter_routing_table_->is_complete); This wasn't quite right if runtime filtering is disabled. Fixed the DCHECK. -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 00:00:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 22 Oct 2019 00:00:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5125/ -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 23:23:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5 parse name() instead
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4849/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 23:26:55 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Fixed typos
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14524 ) Change subject: [DOCS] Fixed typos .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/523/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba Gerrit-Change-Number: 14524 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 22:39:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 10: > Patch Set 10: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5123/ The failure test TestEventProcessing.test_insert_events seems to be unrelated flaky test. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 23:08:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5123/ -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 23:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5 parse name() instead
Xiaomeng Zhang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5_parse_name() instead .. IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5_parse_name() instead We want to use krb5_parse_name() to parse the principal instead of creating our own. As src/kudu/security/init.cc already have g_krb5_ctx initialized, we want to leverage the code in KUDU, and create a wrap up function which can be called from IMPALA Krb5parseName(const string& principal, string* service_name, string* hostname, string* realm) Test done: Add an authentication-test to verify principal with special character. Manually tested with bad format principal, throw out error code 2 instead of original 112 which is BAD_PRINCIPAL_FORMAT error code. Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 --- M be/src/kudu/security/init.cc M be/src/kudu/security/init.h M be/src/rpc/authentication-test.cc M be/src/util/auth-util.cc 4 files changed, 36 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14433/5 -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5 parse name() instead
Xiaomeng Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14433/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14433/4//COMMIT_MSG@7 PS4, Line 7: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead > See comments on commit msg in patch set 2. Sorry, missed this comments. Fixed in new patch. -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 22:46:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Fixed typos
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14524 ) Change subject: [DOCS] Fixed typos .. [DOCS] Fixed typos Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba Reviewed-on: http://gerrit.cloudera.org:8080/14524 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_webui.xml 1 file changed, 3 insertions(+), 3 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba Gerrit-Change-Number: 14524 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4848/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 22:31:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Fixed typos
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14524 Change subject: [DOCS] Fixed typos .. [DOCS] Fixed typos Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba --- M docs/topics/impala_webui.xml 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/14524/1 -- To view, visit http://gerrit.cloudera.org:8080/14524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba Gerrit-Change-Number: 14524 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Fixed typos
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14524 ) Change subject: [DOCS] Fixed typos .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/523/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba Gerrit-Change-Number: 14524 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 22:12:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Fixed typos
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14524 ) Change subject: [DOCS] Fixed typos .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50a7f53e1166bf86bb36b6ad0a1a99fe7a1d89ba Gerrit-Change-Number: 14524 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 22:12:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5125/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 21:46:45 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Fixed a DITA formatting error
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14523 ) Change subject: [DOCS] Fixed a DITA formatting error .. [DOCS] Fixed a DITA formatting error Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 Reviewed-on: http://gerrit.cloudera.org:8080/14523 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_webui.xml 1 file changed, 12 insertions(+), 12 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 Gerrit-Change-Number: 14523 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] Fixed a DITA formatting error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14523 ) Change subject: [DOCS] Fixed a DITA formatting error .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/522/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 Gerrit-Change-Number: 14523 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 21:49:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 21:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/14511/5/be/src/runtime/coordinator-filter-state.h File be/src/runtime/coordinator-filter-state.h: http://gerrit.cloudera.org:8080/#/c/14511/5/be/src/runtime/coordinator-filter-state.h@160 PS5, Line 160: filter_routing_table_ > You missed updating this Done -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 21:46:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14511 to look at the new patch set (#6). Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. IMPALA-9019: fix runtime filter propagation with mt_dop The bug was that filter routing table construction removed filters from the TPlanNode structure for the join when a finstance was not a producer of that filter. The TPlanNode is shared between all instances of a fragment on a backend, so this meant that the filter was removed for all instances on that backend, often meaning that no filters would be produced at all. It was awkward fixing the bug within the framework of the current data structures, where the routing table is keyed by filter_id, so I ended up refactoring the routing table somewhat. This also allowed fixing a TODO about O(n^2) construction of the routing table. Testing: Add regression test that timed out without fix. Perf: Ran a single node TPC-H workload with scale factor 30. No perf change. Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M common/thrift/ImpalaInternalService.thrift M testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test 7 files changed, 131 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14511/6 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 21:46:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Fixed a DITA formatting error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14523 ) Change subject: [DOCS] Fixed a DITA formatting error .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/522/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 Gerrit-Change-Number: 14523 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 21:38:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14398 )
Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table
without HMS config
..
Patch Set 5:
(5 comments)
Thank you for adding the test. I have some concerns about how it changes config
files around, so I proposed an alternative solution.
http://gerrit.cloudera.org:8080/#/c/14398/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14398/5//COMMIT_MSG@7
PS5, Line 7: managerd
spelling: managed (also a few cases below).
http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java
File
fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java:
http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@35
PS5, Line 35: managerd
spelling: managed
http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@38
PS5, Line 38: public class CreateKuduTableWithoutHMSTest {
Usually we write this kind of custom cluster test in Python instead of Java,
except for special cases where the test requires a java library (like LDAP).
I think this is OK and I don't want to ask you to rewrite the test, but just in
case you write one in future, I would use one of the tests under
tests/custom_cluster/ as an example.
http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@66
PS5, Line 66: managerd
spelling: managed
http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@84
PS5, Line 84: String replace_cmd = String.format("mv %s %s",
HIVE_SITE_WITHOUT_HMS, HIVE_SITE);
I don't really like this test modifying global test configs, it really risks
messing up someone's dev environment and having a hard time debugging things.
I think it would be better to put the generated hive-site.xml in a different
directory, e.g. /tmp/CreateKuduTableWithoutHMSTest and then add that to the
classpath for the test.
You would need to prepend it to the CLASSPATH environment variable in
start-daemon.sh. Maybe you could have something like this in start-daemon.sh:
. ${IMPALA_HOME}/bin/set-classpath.sh
if [[ -v TEST_PREPEND_CLASSPATH ]]; then
CLASSPATH=$TEST_PREPEND_CLASSPATH:$CLASSPATH
fi
Then your test could set TEST_PREPEND_CLASSPATH, or pass it as a flag to
start-impala-cluster.py
--
To view, visit http://gerrit.cloudera.org:8080/14398
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297
Gerrit-Change-Number: 14398
Gerrit-PatchSet: 5
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Mon, 21 Oct 2019 21:43:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Fixed a DITA formatting error
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14523 ) Change subject: [DOCS] Fixed a DITA formatting error .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 Gerrit-Change-Number: 14523 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 21:38:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Fixed a DITA formatting error
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14523 Change subject: [DOCS] Fixed a DITA formatting error .. [DOCS] Fixed a DITA formatting error Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 --- M docs/topics/impala_webui.xml 1 file changed, 12 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/14523/1 -- To view, visit http://gerrit.cloudera.org:8080/14523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I63ffeaa16a589bc2c0b3791f25a9d8900587d9f3 Gerrit-Change-Number: 14523 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4847/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 21:01:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/14511/5/be/src/runtime/coordinator-filter-state.h File be/src/runtime/coordinator-filter-state.h: http://gerrit.cloudera.org:8080/#/c/14511/5/be/src/runtime/coordinator-filter-state.h@160 PS5, Line 160: filter_routing_table_ You missed updating this -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 21:09:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Reviewed-on: http://gerrit.cloudera.org:8080/14521 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_webui.xml 1 file changed, 102 insertions(+), 196 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/521/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 21:01:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4846/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 20:55:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4845/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:51:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 12: (4 comments) http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@338 PS12, Line 338: for the scan range nit: remove? http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@344 PS12, Line 344: Note that this takes into account all of the input fragments, : // not just the leftmost because we expect unions to be symmetrical for purposes of : // planning, unlike joins. nit: how about: "Note that step 1 is modified to run on fragments with union nodes, by considering all input fragments and not just the leftmost because we expect unions to be symmetrical for purposes of planning, unlike joins." http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@392 PS12, Line 392: vector scan_hosts; nit: maybe add a Dcheck(scan_node_ids.size() == 1 || has_union) so that we make sure this methods gets another look whenever this changes. http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test: http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@133 PS11, Line 133: > I think this is covered by the tests on l32 and l52 - alltypes has 24 files Got it, thanks for the explanation. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:42:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:40:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/521/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:36:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14521 to look at the new patch set (#2). Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 --- M docs/topics/impala_webui.xml 1 file changed, 102 insertions(+), 196 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14521/2 -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Xiaomeng Zhang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 --- M be/src/kudu/security/init.cc M be/src/kudu/security/init.h M be/src/rpc/authentication-test.cc M be/src/util/auth-util.cc 4 files changed, 36 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14433/4 -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. IMPALA-6501: Optimize count(star) for Kudu scans IMPALA-5036 added an optimisation for count(star) in Parquet scans that avoids materialising dummy rows. This change provides similar optimization for Kudu tables. Instead of materializing empty rows when computing count star, we use the NumRows field from the Kudu API. The Kudu scanner tuple is modified to have one slot into which we will write the num rows statistic. The aggregate function is changed from count to a special sum function that gets initialized to 0. Tests: * Added end-to-end tests ĚŁ* Added planner tests * Run performance tests on tpch.lineitem Kudu table with 25 set as scaling factor, on 1 node, with mt_dop set to 1, just to measure the speedup gained when scanning. Counting the rows before the optimization took around 400ms, and around 170ms after. Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Reviewed-on: http://gerrit.cloudera.org:8080/14347 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-scanner.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test A testdata/workloads/functional-planner/queries/PlannerTest/kudu-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test A testdata/workloads/functional-query/queries/QueryTest/kudu-stats-agg.test M tests/query_test/test_aggregation.py 18 files changed, 583 insertions(+), 92 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 13 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 4: (1 comment) Commit message still needs improvement http://gerrit.cloudera.org:8080/#/c/14433/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14433/4//COMMIT_MSG@7 PS4, Line 7: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead See comments on commit msg in patch set 2. -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 20:24:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 12 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Oct 2019 20:23:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. Patch Set 1: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/520/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:21:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14433/4/be/src/util/auth-util.cc File be/src/util/auth-util.cc: http://gerrit.cloudera.org:8080/#/c/14433/4/be/src/util/auth-util.cc@91 PS4, Line 91: KUDU_RETURN_IF_ERROR(kudu::security::Krb5ParseName(principal, service_name, hostname, realm), line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 20:15:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 21 Oct 2019 20:15:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:13:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Xiaomeng Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. Patch Set 3: (8 comments) Patch set 3 has jenkins test failure "Bad SAM flags in obtain_sam_padata" on CHECK_EQ(krb5_get_default_realm(krb5_ctx, _realm), 0); http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc File be/src/util/auth-util.cc: http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@90 PS2, Line 90: krb5_context krb5_ctx; > Maybe 'krb5_ctx' is a clearer name? Yes, it should be. http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@93 PS2, Line 93: CHECK_EQ(krb5_get_default_realm(krb5_ctx, _realm), 0); > What is the performance impact of these extra allocation/deallocations? It has negative impact on perf. I added this initialization based on comments in init.cc InitKrb5Ctx "Work around the lack of thread safety in krb5_parse_name() by implicitly initializing g_krb5_ctx->default_realm once." http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@97 PS2, Line 97: krb5_error_code code = krb5_parse_name(krb5_ctx, principal.c_str(), ); > You could join with the next line: Done http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@99 PS2, Line 99: *realm = princ->realm.data; > Is there a test that generates TErrorCode::BAD_PRINCIPAL_FORMAT? No, I will add one. http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@100 PS2, Line 100: krb5_data* data = princ->data; > Maybe need to call Done http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@105 PS2, Line 105: krb5_free_principal(krb5_ctx, princ); > I think you can say Good to know! http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@111 PS2, Line 111: > *hostname = data->data; Done http://gerrit.cloudera.org:8080/#/c/14433/2/be/src/util/auth-util.cc@114 PS2, Line 114: > Maybe need to call Done -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Mon, 21 Oct 2019 20:14:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7504 ParseKerberosPrincipal() should use krb5 parse name() instead
Xiaomeng Zhang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14433 ) Change subject: IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead .. IMPALA-7504 ParseKerberosPrincipal() should use krb5_parse_name() instead Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 --- M be/src/util/auth-util.cc 1 file changed, 17 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14433/3 -- To view, visit http://gerrit.cloudera.org:8080/14433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24 Gerrit-Change-Number: 14433 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14515 ) Change subject: IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL. .. IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL. Atlas needs table location to establish lineage between a newly created external table and its table location. The table location information is not available until the createTable catalog op succeeds. After this change, location information is sent to the backend in the TDDLExecResponse message which adds it to the lineage graph. This information is sent only for create external table queries. Testing: Added a test to verify the tableLocation field is populated for a create external table query lineage. Also, modified the lineage.test file to include location information for all lineages. Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f Reviewed-on: http://gerrit.cloudera.org:8080/14515 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/service/client-request-state.cc M be/src/util/lineage-util.h M common/thrift/CatalogService.thrift M common/thrift/LineageGraph.thrift M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/custom_cluster/test_lineage.py 7 files changed, 882 insertions(+), 449 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14515 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f Gerrit-Change-Number: 14515 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/exec/partitioned-hash-join-builder.cc@101 PS4, Line 101: // Skip over filters that are not produced by this instance of the join. > Maybe worth expanding slightly on why this could be the case (I think its j Done http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator-filter-state.h File be/src/runtime/coordinator-filter-state.h: http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator-filter-state.h@153 PS4, Line 153: boost::shared_mutex lock; The locks were just moved from Coordinator, there isn't a change in the protocol. Agree this is confusing though. > we take it as a shared_mutex. Unless I'm missing something, even before your > change we never take this lock exclusively. .lock()/unlock() acquires exclusive ownership of shared_mutex, so lock_guard acquires exclusive ownership. shared_lock acquires shared ownership. That said, this is all counterintuitive and it's not obvious why it's correct so I added a comment where it's acquired and added notes in a few more places. http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator-filter-state.h@156 PS4, Line 156: filter_routing_table_ > update Done http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator.h File be/src/runtime/coordinator.h: http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator.h@386 PS4, Line 386: boost::scoped_ptr > std::unique_ptr? Done. Also updated other instances to make this file consistent. -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:07:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14511 to look at the new patch set (#5). Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. IMPALA-9019: fix runtime filter propagation with mt_dop The bug was that filter routing table construction removed filters from the TPlanNode structure for the join when a finstance was not a producer of that filter. The TPlanNode is shared between all instances of a fragment on a backend, so this meant that the filter was removed for all instances on that backend, often meaning that no filters would be produced at all. It was awkward fixing the bug within the framework of the current data structures, where the routing table is keyed by filter_id, so I ended up refactoring the routing table somewhat. This also allowed fixing a TODO about O(n^2) construction of the routing table. Testing: Add regression test that timed out without fix. Perf: Ran a single node TPC-H workload with scale factor 30. No perf change. Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M common/thrift/ImpalaInternalService.thrift M testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test 7 files changed, 131 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14511/5 -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:06:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5124/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:06:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14521 ) Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/520/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Oct 2019 19:57:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14521 Change subject: IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI .. IMPALA-8970: [DOCS] Documented the Prometheus metrics pages in Web UI Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 --- M docs/topics/impala_webui.xml 1 file changed, 101 insertions(+), 196 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14521/1 -- To view, visit http://gerrit.cloudera.org:8080/14521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifbc68836cc03df3f2a5342eac9230debc4ab1584 Gerrit-Change-Number: 14521 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4844/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 19:25:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4843/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 19:20:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/14422 )
Change subject: IMPALA-8738: Extend "show tables" to return values other than
the table name
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/14422/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14422/4//COMMIT_MSG@17
PS4, Line 17: Testing:
Could you also add a test case to:
- ParserTest.TestShow()
- testdata/workloads/functional-query/queries/QueryTest/show.test, which is run
by tests/metadata/test_metadata_query_statements.py::test_show
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/cup/sql-parser.cup@2465
PS4, Line 2465: {: RESULT = new ShowTablesStmt(false); :}
Did my suggestion to add an 'opt_extended' rule not work for some reason?
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java:
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java@53
PS4, Line 53: true ,the
true, the
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/java/org/apache/impala/service/Frontend.java@815
PS4, Line 815: new ArrayList<>();
Lists.newArrayList()
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/main/java/org/apache/impala/service/Frontend.java@817
PS4, Line 817: new HashSet<>();
Sets.newHashSet()
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java:
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java@234
PS4, Line 234: List tables =
The formatting in this function is pretty ugly now. Not the biggest deal, but
you could improve it, for example by doing something like:
PatternMatcher pattern1 = PatternMatcher.createHivePatternMatcher("*");
and then
.getTableNames("functional", pattern1, USER, false)
http://gerrit.cloudera.org:8080/#/c/14422/4/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java@275
PS4, Line 275: public void TestShowExtendedTableResultsFiltered() throws
ImpalaException {
Since "EXTENDED" doesn't really change anything about the authorization of the
query or the pattern matching, I think its probably not necessary to check all
of these cases with "extended=true", just a single one would be fine.
So, instead of adding this additional test, could you just take a single case
from here, doesn't matter which one as long as its one that actually does
return some table types, and put it in the test above.
For example, by taking lines 279-282 and moving them up to line 241, which a
comment says "Test for SHOW EXTENDED" or similar.
--
To view, visit http://gerrit.cloudera.org:8080/14422
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419
Gerrit-Change-Number: 14422
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Mon, 21 Oct 2019 19:18:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 18:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has uploaded a new patch set (#10) to the change originally created by Vihang Karajgaonkar. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. IMPALA-9030: Handle translated external Kudu tables In HMS 3.1 there is a default tranformer introduced which checks the client capabilities and transforms a table before creating it. Additionally, it also makes sure that any managed table which is created is transactional. If a user creates a managed table which is not transactional, it automatically converts such table as external and sets certain table properties to mark such transformed tables. This presents a problem for managed Kudu tables in Impala since managed and external tables are handled differently in Kudu. Specifically, if a Kudu table is managed, certain operations like drop table, rename table, alter table are performed on the Kudu side along with updating the catalog. If the Kudu table is external, the Kudu operations are skipped and only catalog side operations are performed. When the user creates a managed Kudu table, user expects that drop table, rename table should be updated by Impala automatically in Kudu as well. But since HMS 3 transforms such managed tables into external, currently Impala does not perform the Kudu side operations breaking the semantics for the user. This patch makes changes to Catalog so that it can detect such transformed external tables and perform Kudu side operations similar to what it was doing for managed Kudu table when talking with previous HMS versions. Note that this change is in preparation of bumping up the CDP build which will be done in a separate change. For the current CDP build number the patch is essentially a no-op. Testing: 1. Bumped up the CDP build number in a private build so that the HMS translation logic is pulled in. Ran all the tests. Without the patch there are many Kudu tests which were failing. After the patch none of the Kudu tests fail. There were additional Ranger tests which failed due to the CDP bump but those were unrelated to this patch and should be fixed as part of a separate change when the CDP build number is bumped up. Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 8 files changed, 105 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14397/10 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5123/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 10 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 18:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has uploaded a new patch set (#9) to the change originally created by Vihang Karajgaonkar. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. IMPALA-9030: Handle translated external Kudu tables In HMS 3.1 there is a default tranformer introduced which checks the client capabilities and transforms a table before creating it. Additionally, it also makes sure that any managed table which is created is transactional. If a user creates a managed table which is not transactional, it automatically converts such table as external and sets certain table properties to mark such transformed tables. This presents a problem for managed Kudu tables in Impala since managed and external tables are handled differently in Kudu. Specifically, if a Kudu table is managed, certain operations like drop table, rename table, alter table are performed on the Kudu side along with updating the catalog. If the Kudu table is external, the Kudu operations are skipped and only catalog side operations are performed. When the user creates a managed Kudu table, user expects that drop table, rename table should be updated by Impala automatically in Kudu as well. But since HMS 3 transforms such managed tables into external, currently Impala does not perform the Kudu side operations breaking the semantics for the user. This patch makes changes to Catalog so that it can detect such transformed external tables and perform Kudu side operations similar to what it was doing for managed Kudu table when talking with previous HMS versions. Note that this change is in preparation of bumping up the CDP build which will be done in a separate change. For the current CDP build number the patch is essentially a no-op. Testing: 1. Bumped up the CDP build number in a private build so that the HMS translation logic is pulled in. Ran all the tests. Without the patch there are many Kudu tests which were failing. After the patch none of the Kudu tests fail. There were additional Ranger tests which failed due to the CDP bump but those were unrelated to this patch and should be fixed as part of a separate change when the CDP build number is bumped up. Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 8 files changed, 105 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14397/9 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has posted comments on this change. (
http://gerrit.cloudera.org:8080/14397 )
Change subject: IMPALA-9030: Handle translated external Kudu tables
..
Patch Set 8:
(6 comments)
> Patch Set 8:
>
> (1 comment)
>
> > Patch Set 8: Verified-1
> >
> > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5114/
>
> Some test failures need to fix:
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8384/
> Test Result (3 failures / +3)
> org.apache.impala.analysis.AnalyzeKuduDDLTest.TestCreateExternalKuduTable
> org.apache.impala.planner.PlannerTest.testHbase
> org.apache.impala.planner.PlannerTest.testHbaseNoKeyEstimate
>
> Hao, you can run these java tests locally according to
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+load%2C+run%2C+and+create+new+Impala+tests
> e.g. in $IMPALA_HOME:
> (pushd fe && mvn test -Dtest=AnalyzeKuduDDLTest#TestCreateExternalKuduTable)
> (pushd fe && mvn test -Dtest=PlannerTest#testHbase)
> (pushd fe && mvn test -Dtest=PlannerTest#testHbaseNoKeyEstimate)
>
> Looks like we broke something in HBase Tables. I'll also look into it
> tomorrow.
Thanks Quanlong. I didn't do it as I don't have a suitable environment to set
up Impala (which can takes hours to do), and I think this change is obvious
enough to rely on the Jenkins output.
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@341
PS8, Line 341: // External table cannot have 'external.table.purge'
property set, which is considered
: // equivalent to managed table.
: if (Boolean.parseBoolean(
:
getTblProperties().get(KuduTable.TBL_PROP_EXTERNAL_TABLE_PURGE))) {
: throw new AnalysisException(String.format("Table property
'%s' cannot be set to " +
: "true with an external Kudu table.",
KuduTable.TBL_PROP_EXTERNAL_TABLE_PURGE));
: }
> I think manually changing 'external.table.purge' should not be allowed. We
Good point! Updated.
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/main/java/org/apache/impala/catalog/KuduTable.java
File fe/src/main/java/org/apache/impala/catalog/KuduTable.java:
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@140
PS8, Line 140: public static boolean isSynchronizedTable(
> Sorry, don't need to move these... Only Kudu tables are affected.
Ack
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@505
PS8, Line 505: // Cannot specify the number of replicas for external Kudu
tables
> Comment looks incorrect here. Maybe: External Kudu table is not allowed to
Done
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@506
PS8, Line 506: tab (x int)
> Should not specify columns for creating external Kudu table.
Done
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@507
PS8, Line 507: purges
> typo: purge
Done
http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@508
PS8, Line 508: purges
> same here: purge
Done
--
To view, visit http://gerrit.cloudera.org:8080/14397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36
Gerrit-Change-Number: 14397
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Greg Solovyev
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Mon, 21 Oct 2019 18:35:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9019: fix runtime filter propagation with mt dop
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14511 ) Change subject: IMPALA-9019: fix runtime filter propagation with mt_dop .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/exec/partitioned-hash-join-builder.cc@101 PS4, Line 101: // Skip over filters that are not produced by this instance of the join. Maybe worth expanding slightly on why this could be the case (I think its just because broadcast filters may have been assigned to different instances of the join?) http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator-filter-state.h File be/src/runtime/coordinator-filter-state.h: http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator-filter-state.h@153 PS4, Line 153: boost::shared_mutex lock; So, I'm having a hard time wrapping my head around this locking protocol. The comment says that when doing initialize/destroy, we should take this lock exclusively, but that doesn't seem to be the case - in InitFilterRoutingTable/ReleaseExecResources (I assume the init/destroy its referring to) we take it as a shared_mutex. Unless I'm missing something, even before your change we never take this lock exclusively. We also don't take any lock at all when reading the table, eg. in BackendState::Exec(), which I think could happen concurrently with Coordinator::UpdateFilter(). I think that's fine, cause they're not even accessing the same fields, but its not clear. Anyways, I'm not sure I see any actual race conditions or anything, but seems like this could be improved, or at least made more clear with better comments. http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator-filter-state.h@156 PS4, Line 156: filter_routing_table_ update http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator.h File be/src/runtime/coordinator.h: http://gerrit.cloudera.org:8080/#/c/14511/4/be/src/runtime/coordinator.h@386 PS4, Line 386: boost::scoped_ptr std::unique_ptr? -- To view, visit http://gerrit.cloudera.org:8080/14511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26e3628a982d5d9b8b24eb96b28aff11f8aa6669 Gerrit-Change-Number: 14511 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 18:13:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571: harden QueryEventHook execution
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571: harden QueryEventHook execution
..
Patch Set 28:
(5 comments)
Just a few replies to your useful comments.
Waiting for next patch before doing more thinking
http://gerrit.cloudera.org:8080/#/c/13748/28//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13748/28//COMMIT_MSG@19
PS28, Line 19: See Java Thread.setDaemon(boolean) for what a "daemon thread"
entails.
> I think it depends on whether or not they feel that query hook execution sh
If I'm a user I don't want to read javadoc, I need help deciding. The flag is
not even an advanced flag so sooner or later someone will have to explain the
tradeoffs. This is the first actual change mentioned in the commit message so
it must be important?
http://gerrit.cloudera.org:8080/#/c/13748/28/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/28/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@82
PS28, Line 82: * execute. You then create an executor with a thread pool of
size 1 and a hook
> Flashbacks to my technical-writing course :)
:-)
http://gerrit.cloudera.org:8080/#/c/13748/28/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@182
PS28, Line 182: final ArrayBlockingQueue boundedQueue =
> You mean declare the var as the most general type possible, right? Done.
Yes, thanks
http://gerrit.cloudera.org:8080/#/c/13748/28/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@290
PS28, Line 290: t);
> Yep, in fact it will log the stack trace this way! Does look weird and I al
Thanks
http://gerrit.cloudera.org:8080/#/c/13748/28/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/13748/28/fe/src/main/java/org/apache/impala/service/Frontend.java@1624
PS28, Line 1624:* for {@link QueryEventHook}, which is part of the
published api.
> So the `QueryEventHook` class is published in a separate jar than the rest
Thanks for thinking about this. I can see why the jar (and hence the api) is a
separate artifact.
My question is where the comments will be most useful.
To me it seems like developers are most likely to be reading impala code than
to be reading the api code.
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 28
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Mon, 21 Oct 2019 16:34:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/14080 ) Change subject: IMPALA-8755: Backend support for Z-ordering .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/common/global-flags.cc File be/src/common/global-flags.cc: http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/common/global-flags.cc@275 PS9, Line 275: DEFINE_bool(unlock_zorder_sort, false, : "(Experimental) If true, enables using ZORDER option for SORT BY."); I think we can enable it by default. Or maybe in a follow-up commit, since some tests also need to be moved from custom cluster tests to query tests. http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/util/tuple-row-compare.cc File be/src/util/tuple-row-compare.cc: http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/util/tuple-row-compare.cc@325 PS9, Line 325: ((uint128_t) -1) / 2 + 1 nit: how about (uint128_t)1 << 127? Or you could use SetBit from bit-util.h -- To view, visit http://gerrit.cloudera.org:8080/14080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab Gerrit-Change-Number: 14080 Gerrit-PatchSet: 9 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Oct 2019 16:23:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5121/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 21 Oct 2019 16:00:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 11: Code-Review+1 carry the +1 -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 16:09:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5122/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 12 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Oct 2019 16:06:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 12 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Oct 2019 16:06:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: Yeah looks like it failed to setup the worker for that one test. Hopeful just a once-off infra issue. I restarted the job. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 21 Oct 2019 16:00:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/1 )
Change subject: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
..
Patch Set 2:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/1/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/1/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1854
PS2, Line 1854: tryLock(table, "truncating");
As we have discussed in person, this locking only protects against the specific
case when a table is upgraded in Impala from non-ACID to ACID after the call to
getExistingTable(). I think we generally do not handle this case.
I am ok with either keeping or removing this locking, but if we keep it, then
some comments about its purpose would be nice.
http://gerrit.cloudera.org:8080/#/c/1/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1873
PS2, Line 1873:
Preconditions.checkState(!catalog_.getLock().isWriteLockedByCurrentThread());
This looks a bit dangerous - if we throw an exception when we have the catalog
lock, then it won't be released.
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py
File tests/stress/test_acid_stress.py:
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@164
PS2, Line 164: Task(self._hive_role_write_inserts, tbl_name, partitioned),
nit: here and at other places: doesn't this need indentation of 4?
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@219
PS2, Line 219: tbl_name, wid, num_inserts))
nit: +2 indent
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@223
PS2, Line 223: except Exception as e:
: # Some transactions might fail due to high contention.
: print str(e)
: if "TransactionException" not in str(e):
: raise e
I think it would be better not to swallow these exceptions if possible.
If truncate would get the HMS lock before the catalog's table lock, then the
"big waits" (when TRUNCATE waits for exclusive lock or INSERT's shared lock is
blocked by a TRUNCATE's exclusive lock) would happen when waiting for HMS
locks, which seems more normal than waiting long for catalog locks, and should
have longer timeouts.
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@314
PS2, Line 314: "TransactionException" not in str(e) and
: "CLIENT_REQUEST_UPDATE_CATALOG" not in str(e)
Same as line 223. CLIENT_REQUEST_UPDATE_CATALOG errors could be accepted only
if the debug action was inserted.
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@343
PS2, Line 343: self.client.execute("""create table %s (i int) TBLPROPERTIES
(
Similarly to TestAcidInsertsBasic, I think it would be nice to run the same
tests on partitioned tables, e.g. with writing to different partitions
randomly. This seems like a "cheap" way to cover more code. I am ok with
leaving this as a TODO.
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@353
PS2, Line 353: checker
nit: this could be "checkers", as there can be several of them
http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@354
PS2, Line 354: num_writers)
nit: would fit to last line
--
To view, visit http://gerrit.cloudera.org:8080/1
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d
Gerrit-Change-Number: 1
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 21 Oct 2019 15:16:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14515 ) Change subject: IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL. .. Patch Set 2: The precommit job failed with an unrelated test failure. Filed https://issues.apache.org/jira/browse/IMPALA-9073 for it and re-ran the pre-commit job. Now it succeeded. Tim, could you please merge the patch now? -- To view, visit http://gerrit.cloudera.org:8080/14515 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f Gerrit-Change-Number: 14515 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 21 Oct 2019 15:17:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5120/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 14:02:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4842/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 13:20:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 11 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Oct 2019 13:18:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 8: > Patch Set 6: > > (4 comments) > > Thanks Vihang's comments! > > > I looks like you are sending all the objects in the catalogd when you > > detect the reset. Is it possible to not send the objects and just > > invalidate the whole local catalog cache when you detect that the last > > reset version is changed? > > Reseting coordinator's local catalog is ok. But coordinator needs to know > when the reset() in catalogd finishs. For SYNC_DDL, it also needs to know the > catalog topic version (not catalog version, it's the version of the > statestore topic) so it can wait for other coordinators to be ready. I have > an explanation with code links in "Why can’t we simply reset the cache of > local catalog mode Coordinator for global INVALIDATE METADATA?" in the doc: > https://docs.google.com/document/d/1-AoigzsQPgSGosW4vtVP8E7TiwoJJfLkAq3Rd6KvKBg Ah, I realize what you say is different! It's a good idea to invalidate the whole cache when detecting last reset version is changed! Then we no longer need to change the skipping logics in catalogd, making this patch much simpler. Update the patch according to this. BTW, still contains the fix for IMPALA-9062. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 12:41:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Hello Bharath Vissapragada, Vihang Karajgaonkar, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14307 to look at the new patch set (#8). Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. IMPALA-7506: support global INVALIDATE METADATA in local catalog mode The minimal catalog object version of valid catalog objects is used to implement global invalidate metadata in legacy catalog mode. Coordinator sends DDL RPC to catalogd for global invalidate metadata and gets the expected min catalog version in the response. It's the version when catalogd starts to reset the entire catalog, which means when the reset is done, all valid catalog objects should be associated with a catalog version larger than it. Coordinator will wait until its min catalog version exceeds this value, which means it has processed all the updates of the reset propagated from the catalogd via statestored. If SYNC_DDL is set, the coordinator will also wait until other coordinators reach the same statestore topic version with it, so they have also processed the same updates and had the latest catalog after reset. In local catalog mode, the coordinator does not cache all the metadata. Instead, it caches them on-demand (based on query requests), and removes them based on the Guava cache configurations (size or TTL) or explicit invalidation from the catalog topic updates. So it's hard to track the minimal catalog object version correctly. This patch adds a new field (lastResetCatalogVersion) in TCatalog to propagate the catalog version when catalogd starts to reset the entire metadata. Each time when catalogd generates a new topic update, it will generate a TCatalogObject of CATALOG type containing the state of the catalog which includes this new field. When coordinator receives a new value of lastResetCatalogVersion in a topic update, it means catalogd has reset the entire catalog. Coordinator will then clear its cache to remove all stale catalog objects. It's possible that some fresh items being removed too. They will be refetched on demand. After the invalidation, there are no catalog object cached with catalog version <= lastResetCatalogVersion. Because stale cache has been cleared and all metadata from catalogd is newer than lastResetCatalogVersion. So lastResetCatalogVersion + 1 is the lower bound (included) of min catalog object version of a coordinator. This patch also fixes IMPALA-9062 to avoid catalogd's update collector thread being blocked by concurrent DDLs that holding the table locks. In local catalog mode, we just need to propagate the table name of a changed table, so don't need to acquire table lock to get the entire TTable object. This patch also exposes the min catalog object version of coordinator via a new metric "catalog.min-catalog-object-version" to ease debugging. Tests: - Recover all existing tests that have been disabled due to this missing feature - Run CORE tests Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 --- M be/src/service/impala-server.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/CatalogObjects.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/authorization/test_grant_revoke.py M tests/authorization/test_ranger.py M tests/common/skip.py M tests/custom_cluster/test_local_catalog.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py 14 files changed, 176 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14307/8 -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14080 ) Change subject: IMPALA-8755: Backend support for Z-ordering .. Patch Set 9: (3 comments) http://gerrit.cloudera.org:8080/#/c/14080/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14080/8//COMMIT_MSG@12 PS8, Line 12: The commit adds a Comperator based on Z-ordering. See in detail: Nit: comparator. Also on line 15. http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/exec/partial-sort-node.cc File be/src/exec/partial-sort-node.cc: http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/exec/partial-sort-node.cc@54 PS9, Line 54: sorting_order_ = (TSortingOrder::type)tnode.sort_node.sort_info.sorting_order; I think we're trying to avoid C-style casts. http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/exec/sort-node.cc File be/src/exec/sort-node.cc: http://gerrit.cloudera.org:8080/#/c/14080/9/be/src/exec/sort-node.cc@50 PS9, Line 50: sorting_order_ = (TSortingOrder::type)tnode.sort_node.sort_info.sorting_order; I think we're trying to avoid C-style casts. -- To view, visit http://gerrit.cloudera.org:8080/14080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab Gerrit-Change-Number: 14080 Gerrit-PatchSet: 9 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Oct 2019 11:48:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 8: (2 comments) The test failure in AnalyzeKuduDDLTest is due to a typo. The test failures in testHbase and testHbaseNoKeyEstimate look flaky. Cannot reproduce them in my local env and is not shown in the first GVO. I think it's not relative to this patch. http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java: http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@507 PS8, Line 507: purges typo: purge http://gerrit.cloudera.org:8080/#/c/14397/8/fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java@508 PS8, Line 508: purges same here: purge -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 21 Oct 2019 11:10:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14422 ) Change subject: IMPALA-8738: Extend "show tables" to return values other than the table name .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4840/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 Gerrit-Change-Number: 14422 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 21 Oct 2019 08:46:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14422 ) Change subject: IMPALA-8738: Extend "show tables" to return values other than the table name .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4841/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 Gerrit-Change-Number: 14422 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 21 Oct 2019 08:48:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14422 ) Change subject: IMPALA-8738: Extend "show tables" to return values other than the table name .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4839/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 Gerrit-Change-Number: 14422 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 21 Oct 2019 08:35:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14422 to look at the new patch set (#4). Change subject: IMPALA-8738: Extend "show tables" to return values other than the table name .. IMPALA-8738: Extend "show tables" to return values other than the table name The syntax is "show tables", now adding support for "show extended tables". Show tables only returns the name of the table, and the functionality remains the same.The command show extended tables returns table_type in addition to the table name. Table_type is obtained from the table metadata that has been loaded. So it may take some time if the table has not yet been loaded. Testing: - fe AuthorizationTest#TestShowTableResultsFiltered - fe AuthorizationTest#TestShowExtendTableResultsFiltered Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 --- M be/src/catalog/catalog.cc M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java 12 files changed, 171 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/14422/4 -- To view, visit http://gerrit.cloudera.org:8080/14422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 Gerrit-Change-Number: 14422 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14422 )
Change subject: IMPALA-8738: Extend "show tables" to return values other than
the table name
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14422/2/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java:
http://gerrit.cloudera.org:8080/#/c/14422/2/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java@308
PS2, Line 308: List expectedTableTypes =
Lists.newArrayList("EXTERNAL_TABLE", "EXTERNAL_TABLE",
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14422
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419
Gerrit-Change-Number: 14422
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Mon, 21 Oct 2019 07:51:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14422 to look at the new patch set (#3). Change subject: IMPALA-8738: Extend "show tables" to return values other than the table name .. IMPALA-8738: Extend "show tables" to return values other than the table name The syntax is "show tables", now adding support for "show extended tables". Show tables only returns the name of the table, and the functionality remains the same.The command show extended tables returns table_type in addition to the table name. Table_type is obtained from the table metadata that has been loaded. So it may take some time if the table has not yet been loaded. Testing: - fe AuthorizationTest#TestShowTableResultsFiltered - fe AuthorizationTest#TestShowExtendTableResultsFiltered Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 --- M be/src/catalog/catalog.cc M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-http-handler.cc M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java 12 files changed, 171 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/14422/3 -- To view, visit http://gerrit.cloudera.org:8080/14422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419 Gerrit-Change-Number: 14422 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8738: Extend "show tables" to return values other than the table name
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14422 )
Change subject: IMPALA-8738: Extend "show tables" to return values other than
the table name
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14422/3/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java:
http://gerrit.cloudera.org:8080/#/c/14422/3/fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java@308
PS3, Line 308: List expectedTableTypes =
Lists.newArrayList("EXTERNAL_TABLE", "EXTERNAL_TABLE",
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14422
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63057c9d2fc453f95c6890bdc90e11c61a98a419
Gerrit-Change-Number: 14422
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Mon, 21 Oct 2019 08:02:13 +
Gerrit-HasComments: Yes
