[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.h File be/src/runtime/io/data-cache.h: http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.h@88 PS2, Line 88: cosolidating consolidating http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.h@89 PS2, Line 89: sprase sparse http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.h@184 PS2, Line 184: emtpy empty http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/hdfs-file-reader.cc File be/src/runtime/io/hdfs-file-reader.cc: http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/hdfs-file-reader.cc@193 PS1, Line 193: return Status(TErrorCode::DISK_IO_ERROR, GetBackendString(), > Addressed the concern with larger entry in the new patch set. To answer the Two cases I can think of: * If a scan was not able to get its "ideal" reservation and is scanning with a smaller buffer size. This is hopefully rare. * With the new parquet page index support if different subset of pages are selected. I'm ok with deferring this but I think it is worth understanding if there's a bad interaction between this and page indexes. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 06:58:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
Hello Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12939
to look at the new patch set (#4).
Change subject: IMPALA-8386: Fix incorrect equivalence conjuncts not treated as
identity
..
IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
When generating single node plans for inline views, Impala will create
some equivalence conjuncts based on slot equivalences. However, these
conjuncts may finally be substituted to identity (e.g. a = a) which may
incorrectly reject rows with nulls. We already have some logic to remove
this kind of conjuncts but the existing checks have exceptions.
For example, consider the following tables and a query:
table Atable Btable C
+--+ +--++ +--+--+
| a_id | | b_id | amount | | a_id | b_id |
+--+ +--++ +--+--+
| 1| | 1| 10 | | 1| 1|
| 2| | 1| 20 | | 2| 2|
+--+ | 2| NULL | +--+--+
+--++
select * from (select t2.a_id, t2.amount1, t2.amount2
from a
left outer join (
select c.a_id, amount as amount1, amount as amount2
from b join c on b.b_id = c.b_id
) t2
on a.a_id = t2.a_id
) t1;
They query has 11 slots. The valueTransferGraph (slot equivalences) has
3 strongly connected components:
* {slot0 (b.b_id), slot1 (c.b_id)}
* {slot2 (c.a_id), slot4 (t2.a_id), slot8 (t1.a_id)}
* {slot3 (b.amount), slot5 (t2.amount1), slot6 (t2.amount2),
slot9 (t1.amount1), slot10 (t1.amount2)}
In SingleNodePlanner#migrateConjunctsToInlineView, when dealing with
inline view t1, a predicate "t1.amount1 = t1.amount2" will first be
created by Analyzer#createEquivConjuncts, then be substituted using the
smap_ of the inline view and become "t2.amount1 = t2.amount2". It can
still pass the IdentityPredicate check. However, the substituted one
will finally be resolved to "amount = amount" and be assigned to the
left outer join node. So nulls are incorrectly rejected.
Actually, when checking IdentityPredicates, we need to check the final
resolved version of them using base table slots (baseTblSmap_). So the
predicate "t1.amount1 = t1.amount2" will be resolved to "amount = amount"
and won't pass the IdentityPredicate check.
Tests:
* Add plan tests in PlannerTest/inline-view.test
* Run all tests locally in CORE exploration strategy
Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
3 files changed, 221 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/12939/4
--
To view, visit http://gerrit.cloudera.org:8080/12939
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa
Gerrit-Change-Number: 12939
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2813/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 06:46:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Fang-Yu Rao has removed Anurag Mantripragada from this change. ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. Removed reviewer Anurag Mantripragada. -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2805/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 18 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 21:43:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4032/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 06:13:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.cc File be/src/runtime/io/data-cache.cc: http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.cc@147 PS1, Line 147: if (charge_len > capacity_) return false; > Given that we won't actually insert into the cache until after trying to wr This actually seems to be a problematic behavior. We may temporarily exceed the capacity limit as a result of this. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 05:57:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. IMPALA-8392: fix parallel docker_images build I made the other targets depend on targets, not the timestamp file, according to the suggested solution in: https://gitlab.kitware.com/cmake/cmake/issues/17585 Testing: Ran "make -j 8 docker_images" locally, which now succeeds. Running dockerised tests. Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce --- M bin/jenkins/dockerized-impala-run-tests.sh M docker/CMakeLists.txt 2 files changed, 11 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/13053/2 -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4031/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Apr 2019 05:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 05:30:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. The patch also fixes some TODOs to replace the rangerPlugin.init() hack with rangerPlugin.refreshPoliciesAndTags() API available in this Ranger build. Testing: - Ran core tests - Manually verified that no regression when starting Hive 3 with USE_CDP_HIVE=true Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Reviewed-on: http://gerrit.cloudera.org:8080/13002 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M .gitignore M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/mvn-quiet.sh M buildall.sh M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 10 files changed, 84 insertions(+), 227 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2807/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 22:06:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7995: print webpage to debug test failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13052 ) Change subject: IMPALA-7995: print webpage to debug test failure .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2812/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0425aa9cb173548e47377c5738c0cb070c21fd87 Gerrit-Change-Number: 13052 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 04:47:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2811/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 04:36:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 11: Code-Review+1 Thanks for making the changes and testing that HMS3 comes up. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 22:50:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. IMPALA-8415: Fix tests broken by storage layer information Storage layer information was added to the query profile by IMPALA-6050. This broke some tests on exhaustive and s3 runs due to changes in formatting. This fixes the issues: 1. Replace HDFS SCAN with $FILESYSTEM_NAME SCAN in some test files 2. Add $FILESYSTEM_NAME to partition information string Testing: - Ran exhaustive HDFS tests - Ran s3 tests Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Reviewed-on: http://gerrit.cloudera.org:8080/13025 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/corrupt-stats.test M testdata/workloads/functional-query/queries/QueryTest/set.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/metadata/test_ddl.py 4 files changed, 36 insertions(+), 30 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 6: Thanks for your first contribution! -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 03:56:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7995: print webpage to debug test failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13052 ) Change subject: IMPALA-7995: print webpage to debug test failure .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4030/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0425aa9cb173548e47377c5738c0cb070c21fd87 Gerrit-Change-Number: 13052 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 04:04:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7995: print webpage to debug test failure
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13052 ) Change subject: IMPALA-7995: print webpage to debug test failure .. Patch Set 1: Code-Review+2 This is so trivial I'm just going to +2 myself -- To view, visit http://gerrit.cloudera.org:8080/13052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0425aa9cb173548e47377c5738c0cb070c21fd87 Gerrit-Change-Number: 13052 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 04:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. IMPALA-8375: Add metrics for spill disk usage Added two new metrics tmp-file-mgr.scratch-space-bytes-used-high-water-mark & tmp-file-mgr.scratch-space-bytes-used for tracking HWM and current value for spilled bytes, respectively. A new class AtomicHighWaterMarkGauge was added to keep track of the HWM value. The new class also encapsulates a metric object which keeps track of the current value for the spilled bytes. The current value is incremented every time a new range is allocated from a temporary file. The current value for spilled bytes is decremented when a temporary file is closed. The new metrics are not updated when ranges are recycled from a file. We can add a new metric in future for keeping track of actual spilled bytes. The HWM value is updated whenever the current value is greater than the HWM value. Testing: - Added new unit tests to the metrics-test test case. - E2E testing for both the metrics by running concurrent spilling queries and ensuring that both the current value metric and the HWM metric were behaving as expected. Ran concurrent queries and monitored the metrics on the impala daemon's metric page. Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Reviewed-on: http://gerrit.cloudera.org:8080/12956 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/CMakeLists.txt M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/metrics-test.cc M be/src/util/metrics.h M common/thrift/metrics.json 6 files changed, 135 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7995: print webpage to debug test failure
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13052 Change subject: IMPALA-7995: print webpage to debug test failure .. IMPALA-7995: print webpage to debug test failure TestWebPage::test_catalog failed on a test run. Printing the response test will help debug in future. Change-Id: I0425aa9cb173548e47377c5738c0cb070c21fd87 --- M tests/webserver/test_web_pages.py 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/13052/1 -- To view, visit http://gerrit.cloudera.org:8080/13052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0425aa9cb173548e47377c5738c0cb070c21fd87 Gerrit-Change-Number: 13052 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Fang-Yu Rao has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. IMPALA-4865: Reject Expr Rewrite When Appropriate Avoided rewrite if the resulting string literal exceeds a defined limit. Testing: Added three statements in testFoldConstantsRule() to verify that the expression rewrite is accepted only when the size of the rewritten expression is below a specified threshold. Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 --- M .gitignore M be/src/service/fe-support.cc M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java M fe/src/main/java/org/apache/impala/analysis/PartitionKeyValue.java M fe/src/main/java/org/apache/impala/analysis/RangePartition.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/rewrite/FoldConstantsRule.java M fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java 13 files changed, 114 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/12814/8 -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 6: Please close the JIRA and set the "Fix version" when you have a chance. -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 03:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7957: Fix slot equivalences may be enforced multiple times
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13051 ) Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2810/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Apr 2019 03:52:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2808/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 23:30:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7957: Fix slot equivalences may be enforced multiple times
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13051 Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times .. IMPALA-7957: Fix slot equivalences may be enforced multiple times Predicates can be divided into three types according to the way they are generated: 1) origin predicates that come from the query 2) auxiliary equal predicates generated for equivalence between a label(alias) and its real expression 3) inferred predicates that inferred from the slot equivalences graph The slot equivalences graph (valueTransferGraph in Analyzer) is generated by the first two kinds of predicates. Analyzer will create equivalence predicates for a PlanNode based on the unassigned predicates and the valueTransferGraph. However, the current implementation can't avoid creating inferred predicates that are duplicated with previously created inferred predicates if they have been assigned before. Duplicated inferred predicates are either redundant or wrong. Say, if we create predicate p1: s1 = s2 for the current PlanNode and p1 duplicates with a previously inferred predicate p0: s1 = s2 (same as s2 = s1), we can prove that p1 is redundant or wrong: 1) p0 must have been assigned. Otherwise, p0 will be in the unassigned conjuncts list and p1 won't be created. 2) p0 must have been assigned to an offspring node of the current PlanNode since we create the PlanNodes in a depth first manner. 3) The origin predicates that infer to p0 have been assigned to an offspring node too. Then, rows that should be rejected have been filtered out either by p0 or the origin predicates that infer to p0. What's worse, assigning p1 on top of the origin predicates may wrongly reject rows. Hence, p1 is either redundant or wrong. In this patch, we check the existence of previously inferred equivalence predicates before creating a predicate. Also add some useful TRACE level logs. Tests: * Add tests for UNIONs in inline-view.test * Run all tests locally in CORE exploration strategy Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test 5 files changed, 292 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/13051/2 -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 03:19:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2804/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 21:20:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2809/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 01:06:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4028/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 22:09:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2798/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 18:15:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12987 )
Change subject: IMPALA-8341: Data cache for remote reads
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.cc
File be/src/runtime/io/data-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.cc@296
PS2, Line 296: if (pending_insert_set_.size() >=
FLAGS_data_cache_write_concurrency ||
: pending_insert_set_.find(key.ToString()) !=
pending_insert_set_.end()) {
: return false;
: }
TODO: Add a metric for this.
--
To view, visit http://gerrit.cloudera.org:8080/12987
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc
Gerrit-Change-Number: 12987
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Ho
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 16 Apr 2019 19:47:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4029/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 00:15:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/Db.java File fe/src/main/java/org/apache/impala/catalog/Db.java: http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/Db.java@524 PS1, Line 524: + "could cause unnecessary database invalidation when the event is processed", line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 00:23:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
xiaom...@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13049 Change subject: IMPALA-8149 : Add support for alter_database events .. IMPALA-8149 : Add support for alter_database events This change add support for alter_database events in two parts: One is adding catalogServiceId and catalogVersion in db parameters when alter database. The other is adding alter database event, check if it's self event during process, if true do nothing, if false replace caralog cached db with event db. Testing: Enabled testAlterDisableFlagFromDb in MetastoreEventsProcessorTest. Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 254 insertions(+), 102 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13049/1 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 11: Code-Review+2 Bump this to +2 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 00:14:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 11: Code-Review+1 > Patch Set 10: > > Hi Fredy, was wondering if you were able to start minicluster which spins up > HMS3 using this build? Just wanted to make sure that we are not regressing > IMPALA-8345 until we have automated jobs running against hive-3 I tested starting the minicluster with Hive 3 and no problem so far. I updated the commit message. Carry Joe's +1. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 22:49:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. The patch also fixes some TODOs to replace the rangerPlugin.init() hack with rangerPlugin.refreshPoliciesAndTags() API available in this Ranger build. Testing: - Ran core tests - Manually verified that no regression when starting Hive 3 with USE_CDP_HIVE=true Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M .gitignore M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/mvn-quiet.sh M buildall.sh M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 10 files changed, 84 insertions(+), 227 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/11 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 22:40:12 + Gerrit-HasComments: No
[Impala-ASF-CR] Initial support for recursive file listing within a partition
Sudhanshu Arora has posted comments on this change. ( http://gerrit.cloudera.org:8080/12991 ) Change subject: Initial support for recursive file listing within a partition .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12991/1/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/12991/1/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@154 PS1, Line 154: FlatBufferBuilder fbb = new FlatBufferBuilder(1); Why remove the Precondition? -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 22:26:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12956/4/be/src/util/metrics.h File be/src/util/metrics.h: http://gerrit.cloudera.org:8080/#/c/12956/4/be/src/util/metrics.h@254 PS4, Line 254: /// maintains the current value. Note that since two separate atomics are used > I have made the suggested change. I was looking for a standard way to docum I'm open to the idea. We use the three-slash comments so that they're picked up by doxygen but don't use any other features. Some of the earlier authors didn't particularly like overly-structured javadoc-style comments - sometimes people get over-zealous and start over-documenting obvious parameters and return values. d...@impala.apache.org is a good place to float ideas like this, in the past we've made changes to the style guide based on informal consensus on the dev list and implemented them incrementally as code was touched. -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 22:15:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 22:09:42 + Gerrit-HasComments: No
[native-toolchain-CR] Enable reusing ccache directories.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: Code-Review+1 (2 comments) Just a few points on readability http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh@519 PS3, Line 519: local TAR=ccache.tar Handling of the tarball filename is asymmetric between download_ccache() and upload_ccache(): it is abstracted here (maybe suggesting that it can be changed), but hardcoded in upload_ccache(). Would be easier to read if it followed the same convention. OTOH if there's a reason for the asymmetry, then a comment could clarify it. http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh@520 PS3, Line 520: S3_URL="https://native-toolchain Bucket name is hardcoded here, but configurable in upload_ccache(). Would be nice if it were symetrical. I realize that other places hardcode the bucket name too, but these two functions are obviously related. -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 16 Apr 2019 21:56:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [acid] Disallow any operation on full acid table.
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13047 )
Change subject: [acid] Disallow any operation on full acid table.
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13047/1/fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java
File fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java:
http://gerrit.cloudera.org:8080/#/c/13047/1/fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java@86
PS1, Line 86: if (tableTransactionalProp == null ||
!tableTransactionalProp.equalsIgnoreCase("insert_only")) {
line too long (102 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb92e5b691bf192980f2cc7d68c491bb1e8455ac
Gerrit-Change-Number: 13047
Gerrit-PatchSet: 1
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 16 Apr 2019 21:28:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Abhishek Rawat has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. IMPALA-8375: Add metrics for spill disk usage Added two new metrics tmp-file-mgr.scratch-space-bytes-used-high-water-mark & tmp-file-mgr.scratch-space-bytes-used for tracking HWM and current value for spilled bytes, respectively. A new class AtomicHighWaterMarkGauge was added to keep track of the HWM value. The new class also encapsulates a metric object which keeps track of the current value for the spilled bytes. The current value is incremented every time a new range is allocated from a temporary file. The current value for spilled bytes is decremented when a temporary file is closed. The new metrics are not updated when ranges are recycled from a file. We can add a new metric in future for keeping track of actual spilled bytes. The HWM value is updated whenever the current value is greater than the HWM value. Testing: - Added new unit tests to the metrics-test test case. - E2E testing for both the metrics by running concurrent spilling queries and ensuring that both the current value metric and the HWM metric were behaving as expected. Ran concurrent queries and monitored the metrics on the impala daemon's metric page. Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 --- M be/CMakeLists.txt M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/metrics-test.cc M be/src/util/metrics.h M common/thrift/metrics.json 6 files changed, 135 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/12956/5 -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [acid] Disallow any operation on full acid table.
Sudhanshu Arora has uploaded this change for review. (
http://gerrit.cloudera.org:8080/13047
Change subject: [acid] Disallow any operation on full acid table.
..
[acid] Disallow any operation on full acid table.
Added a simple check to disallow any operation on fully acid table.
Testing Done:
- I still need to figure out how to add a test case for
this.
- Executed the following sql on mini-cluster
>CREATE TABLE tm (a int, b int) TBLPROPERTIES
>('transactional'='true',
>'transactional_properties'='insert_only');
>select * from tm;
>CREATE TABLE tm2 (a int, b int) stored as orc TBLPROPERTIES
>('transactional'='true');
>Select * from tm2;
The above select failed with Analysis exception.
Change-Id: Ifb92e5b691bf192980f2cc7d68c491bb1e8455ac
---
M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java
1 file changed, 20 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/13047/1
--
To view, visit http://gerrit.cloudera.org:8080/13047
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifb92e5b691bf192980f2cc7d68c491bb1e8455ac
Gerrit-Change-Number: 13047
Gerrit-PatchSet: 1
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. The patch also fixes some TODOs to replace the rangerPlugin.init() hack with rangerPlugin.refreshPoliciesAndTags() API available in this Ranger build. Testing: - Ran core tests Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M .gitignore M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/mvn-quiet.sh M buildall.sh M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 10 files changed, 84 insertions(+), 227 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/10 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 10: Code-Review+1 (1 comment) This looks fine to me. I can +2 once other reviewers have a chance to look. http://gerrit.cloudera.org:8080/#/c/13002/10/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/13002/10/bin/bootstrap_toolchain.py@420 PS10, Line 420: download_cdp_components This is getting very similar to download_cdh_components. At some point, we should consider combining them. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 20:46:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 18: Thanks for the pointers Bharath. I tried to make the wait for event processing more deterministic by scraping the /events page for last_sync_event_id to know if the events have been processed. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 18 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 20:59:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 569 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/18 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 18 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2802/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 20:26:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8329: Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: IMPALA-8329: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 10: This is ready for review. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 20:38:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 2: TODO: - add a test to rotate the files in data-cache-test.cc - add a test for the changes in filesystem-util.cc -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 19:46:40 + Gerrit-HasComments: No
[Impala-ASF-CR] XXX
Michael Ho has abandoned this change. ( http://gerrit.cloudera.org:8080/13026 ) Change subject: XXX .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/13026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ia3ede43146006f7e76707f1b7c8c99aae6671db8 Gerrit-Change-Number: 13026 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Hello Thomas Marshall, Lars Volker, David Rorke, Sahil Takiar, Todd Lipcon, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12987 to look at the new patch set (#2). Change subject: IMPALA-8341: Data cache for remote reads .. IMPALA-8341: Data cache for remote reads This is a patch based on PhilZ's prototype: https://gerrit.cloudera.org/#/c/12683/ This change implements an IO data cache which is backed by local storage. It implicitly relies on the OS page cache management to shuffle data between memory and the storage device. This is useful for caching data read from remote filesystems (e.g. remote HDFS data node, S3, ABFS, ADLS). A data cache is divided into one or more partitions based on the configuration string which is a list of directories, separated by comma, followed by the storage capacity per directory. An example configuration string is like the following: --data_cache_config=/data/0,/data/1:150GB In the configuration above, the cache may use up to 300GB of storage space, with 150GB max for /data/0 and /data/1 respectively. Each partition has a meta-data cache which tracks the mappings of cache keys to the locations of the cached data. A cache key is a tuple of (file's name, file's modification time, file offset) and a cache entry is a tuple of (backing file, offset in the backing file, length of the cached data, optional checksum). Note that the cache currently doesn't support overlapping ranges. In other words, if the cache contains an entry of a file for range [m, m+4MB), a lookup for [m+4K, m+8K) will miss in the cache. In practice, we haven't seen this as a problem but this may require further evaluation in the future. Each partition stores its set of cached data in backing files created on local storage. When inserting new data into the cache, the data is appended to the current backing file in use. The storage consumption of each cache entry counts towards the quota of that partition. When a partition reaches its capacity, the least recently used (LRU) data in that partition is evicted. Evicted data is removed from the underlying storage by punching holes in the backing file it's stored in. As a backing file reaches a certain size (by default 4TB), new data will stop being appended to it and a new file will be created instead. Note that due to hole punching, the backing file is actually sparse. Optionally, checksumming can be enabled to verify read from the cache is consistent with what was inserted and to verify that multiple attempted insertions with the same cache key have the same cache content. Checksumming is enabled by default for debug builds. To probe for cached data in the cache, the interface Lookup() is used; To insert data into the cache, the interface Store() is used. Please note that eviction happens inline currently during Store(). This patch also added two startup flags for start-impala-cluster.py: '--data_cache_dir' specifies the base directory in which each Impalad creates the caching directory '--data_cache_size' specifies the capacity string for each cache directory. Testing done: a new BE test was added; core test with cache enabled. Perf: - 16-streams TPCDS at 3TB in a 20 node S3 cluster shows about 30% improvement over runs without the cache. Each node has a cache size of 150GB per node. The performance is at parity with a configuration of a HDFS cluster using EBS as the storage. Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/data-cache-test.cc A be/src/runtime/io/data-cache.cc A be/src/runtime/io/data-cache.h M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/hdfs-file-reader.h M be/src/runtime/io/request-context.h M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M bin/start-impala-cluster.py M common/thrift/metrics.json 17 files changed, 1,467 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/12987/2 -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2801/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 19:33:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12987 )
Change subject: IMPALA-8341: Data cache for remote reads
..
Patch Set 1:
(71 comments)
This new patch changes the configuration string to use a uniform capacity quota
for all directories.
http://gerrit.cloudera.org:8080/#/c/12987/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12987/1//COMMIT_MSG@24
PS1, Line 24: is a tuple of (file's name, file's modification time, file offset)
> So this means that you can get a partial cache hit if the offsets match but
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/exec/hdfs-scan-node-base.cc@948
PS1, Line 948:
data_cache_miss_bytes_->Set(reader_context_->data_cache_miss_bytes());
> I think for the above counters we figured that this pattern of copying the
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc
File be/src/runtime/io/data-cache-test.cc:
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@52
PS1, Line 52: // The callback is invoked by each thread in the multi-threaded
tests below.
> callback reads like it's called when something is done, how about ThreadFn?
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@157
PS1, Line 157: // Read the same same range inserted previously and they
should still all in the cache.
> nit: be
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@196
PS1, Line 196: // Create a buffer way larger than the cache.
> Why does it need to be larger?
Typo. Fixed.
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@201
PS1, Line 201: ASSERT_TRUE(cache.Store("foobar", 12345, 0,
large_buffer.get(), cache_size));
> Use variables instead of inline constants?
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@220
PS1, Line 220: const int64_t cache_size = 1 << 22;
> I find these easier to read in the form of 4 * 1024 * 1024
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@248
PS1, Line 248: differen
> typo
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@301
PS1, Line 301: ootprint) {
> Can you add a comment what this test does?
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache-test.cc@330
PS1, Line 330: int main(int argc, char **argv) {
> This is not needed anymore if you add your test to the unified backend test
Prefer to keep this test as stand-alone to make development easier.
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h
File be/src/runtime/io/data-cache.h:
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@18
PS1, Line 18: #ifndef IMPALA_RUNTIME_IO_DATA_CACHE_H
> Could use #pragma once instead of include guards
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@53
PS1, Line 53: /// punching holes in the backing file it's stored in. As a
backing file reaches a certain
> It's actually a TODO item to retire older files and copy what's left in the
Added a check for filesystem support for hole punching in the new patch.
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@55
PS1, Line 55: /// created instead. Note that due to hole punching, the backing
file is actually sparse.
> This might be a case where a simple ASCII diagram would illustrate the conc
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@56
PS1, Line 56: ///
> It's by design that we don't support overlapping range for the first implem
Added some explanation in the header comment.
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@63
PS1, Line 63: /// happens inline currently during Store().
> It's worth documenting the PAGE_SIZE behaviour since it implies that there'
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@76
PS1, Line 76: class DataCache{
> style nit: missing space before {
Done
http://gerrit.cloudera.org:8080/#/c/12987/1/be/src/runtime/io/data-cache.h@81
PS1, Line 81: DataCache(const std::string& config) : config_(config) { }
> totally doesn't matter here, but general best practice in C++11 is to have
Thanks for the reminder. Almost every time, I have to look up in the internet
for the advantage of pass-by-value-then-move over pass-by-reference. It's
subtle but it makes sense. We should probably start sticking to this idiom more
widely in Impala code base. May be a clang-tidy rule will help ?!
Also totally irrelevant but that's an area where I find passing by pointer in C
is sometimes easier to use or understand than C++.
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/12987/2/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/12987/2/bin/start-impala-cluster.py@117 PS2, Line 117: ; flake8: E703 statement ends with a semicolon http://gerrit.cloudera.org:8080/#/c/12987/2/bin/start-impala-cluster.py@120 PS2, Line 120: ; flake8: E703 statement ends with a semicolon -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 19:41:39 + Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/3 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. Testing: - Ran core tests Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M .gitignore M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/mvn-quiet.sh M buildall.sh M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 9 files changed, 81 insertions(+), 221 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/8 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 8: Updated .gitignore. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 18:48:09 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. Testing: - Ran core tests Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/mvn-quiet.sh M buildall.sh M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 8 files changed, 80 insertions(+), 221 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/7 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2799/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 18:39:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2795/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 18:09:05 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. Testing: - Ran core tests Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/mvn-quiet.sh M buildall.sh M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 8 files changed, 80 insertions(+), 221 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/6 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2796/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 18:09:17 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/13002/5/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/13002/5/bin/bootstrap_toolchain.py@426 PS5, Line 426: set.") > nit, Since CDP_COMPONENTS_HOME is a env variable, it is perhaps more correc Done http://gerrit.cloudera.org:8080/#/c/13002/5/impala-parent/pom.xml File impala-parent/pom.xml: http://gerrit.cloudera.org:8080/#/c/13002/5/impala-parent/pom.xml@114 PS5, Line 114: impala.cdp.repo > I noticed, that adding such a repository generates a lot of warnings during Good idea. Done. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 18:15:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2794/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 18:07:56 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2797/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 18:09:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 3: (3 comments) Before more work goes into this, this is changing the Hive major version number and there should be a discussion on dev@ about this approach and how it fits into Impala releases. http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@182 PS2, Line 182: if [ -d "$IMPALA_HOME/thirdparty" ]; then : NO_THIRDPARTY=false : else : NO_THIRDPARTY=true : fi We don't use thirdparty anymore, so let's kill it off. DOWNLOAD_CDH_COMPONENTS should default to true (unless set from the environment). http://gerrit.cloudera.org:8080/#/c/13005/2/bin/impala-config.sh@230 PS2, Line 230: . "$IMPALA_HOME/bin/impala-config-branch.sh" : if [ -f "$IMPALA_HOME/bin/impala-config-local.sh" ]; then : . "$IMPALA_HOME/bin/impala-config-local.sh" : fi We need to be careful about which variables are assigned before this and which are assigned after this. A common thing is for developers to customize IMPALA_TOOLCHAIN (needed for distcc). For example, my bin/impala-config-local.sh has: export IMPALA_TOOLCHAIN=/opt/Impala-Toolchain http://gerrit.cloudera.org:8080/#/c/13005/2/shaded-deps/pom.xml File shaded-deps/pom.xml: http://gerrit.cloudera.org:8080/#/c/13005/2/shaded-deps/pom.xml@24 PS2, Line 24: ../../impala-parent/pom.xml Think this should be ../impala-parent/pom.xml -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 17:57:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@199
PS3, Line 199: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@203
PS3, Line 203: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@212
PS3, Line 212: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/3/bin/impala-config.sh@546
PS3, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin/src/standalone-metastore/src/main/thrift
line too long (133 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436
Gerrit-Change-Number: 13005
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 16 Apr 2019 17:54:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/13002/5/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/13002/5/bin/bootstrap_toolchain.py@426 PS5, Line 426: present nit, Since CDP_COMPONENTS_HOME is a env variable, it is perhaps more correct to say make sure it is set than is present. http://gerrit.cloudera.org:8080/#/c/13002/5/impala-parent/pom.xml File impala-parent/pom.xml: http://gerrit.cloudera.org:8080/#/c/13002/5/impala-parent/pom.xml@114 PS5, Line 114: impala.cdp.repo I noticed, that adding such a repository generates a lot of warnings during build since maven tries to find the artificates in cdp repo first and then goes to cdh repo. While this is not a issue per say, it would be good to find a way to fix these warnings if possible or update the mvn-quiet.sh to ignore such warnings from being printed. -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 17:47:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12297 )
Change subject: IMPALA-8138: Reintroduce rpc debugging options
..
Patch Set 1:
(6 comments)
Thanks for working on this patch. My general comment is to consider moving away
from the existing "fake" fault injection framework in Thrift and use debug
actions to simulate scenarios in which we may actually fail to exercise the
entire RPC stack better.
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/rpc/impala-control-service-proxy.h
File be/src/rpc/impala-control-service-proxy.h:
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/rpc/impala-control-service-proxy.h@44
PS5, Line 44:
As mentioned elsewhere, this kind of artificial fault injection doesn't seem to
be too useful.
http://gerrit.cloudera.org:8080/#/c/12297/3/be/src/rpc/impala-control-service-proxy.h
File be/src/rpc/impala-control-service-proxy.h:
http://gerrit.cloudera.org:8080/#/c/12297/3/be/src/rpc/impala-control-service-proxy.h@43
PS3, Line 43:
Not needed. Same below.
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/rpc/rpc-mgr.inline.h
File be/src/rpc/rpc-mgr.inline.h:
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/rpc/rpc-mgr.inline.h@65
PS5, Line 65:
:
:
:
:
:
:
:
:
:
:
:
:
It appears that this send vs recv debug actions were carried over from Thrift
implementation.
Retrospectively, the "fault injection" we did with Thrift was quite hacky (I am
the culprit here) and it stemmed from the total lack of fault injection
testing back then for exercising the error paths in Thrift RPC.
As part of the KRPC development, we invested in proper fault injection testing
by truly pausing the Impala and artificially creates various failure scenario.
This allows a more extensive exercise across the entire RPC stack instead of
just exercising the RPC handlers at the client and server sides.
With the fault injection framework, it seems to be not too useful to continue
with this path of artificial fault injection via debug action we used to do
with Thrift.
Instead, we may want to rethink the fault injection testing with KRPC. In
particular, it may exercise the code better by doing some of the followings:
- use debug actions to inject random delays in the RPC handlers. This is
particularly useful for RPCs with timeout
- use debug actions to randomly reject some of the incoming RPCs in
ImpalaServicePool
- use debug actions to respond with error status in the RPC handlers. The
errors will be specific to each RPC handler (e.g. deserialization error of
Thrift profiles, deserialization errors of RowBatch)
- debug action to force some incoming RPCs to use deferred queue in
KrpcDataStreamRecvr
- (experimental / dangerous) "randomly" corrupt the incoming RPC payloads in
ImpalaServicePool.
- inject delay in RPC callback in the client side to simulate an overloaded
client
The above are some examples I can think of right now.
For other failures, we may need to rely on the fault injection framework:
- use iptables to drop all incoming packets to the RPC port from a particular
host. This simulates a host which was powered off or network partitions
- Restart remote Impalad will trigger the behavior of broken connections (by
sending a RST packet)
- Send SIGSTOP to remote Impalad (which we already do in the fault injection
framework) to simulate non-responsive Impalad
- other ideas...
In general, my suggestion is to use debug actions to simulate failure which can
actually happen instead of using this artificial fault injection which seems a
bit meaningless at this point.
http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/query-state.h
File be/src/runtime/query-state.h:
http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/query-state.h@314
PS1, Line 314: proxy_
> I assume that if I change the class name back to something ending in "Proxy
Yup.
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/service/control-service.cc
File be/src/service/control-service.cc:
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/service/control-service.cc@171
PS5, Line 171: if (qs.get() == nullptr) {
: Status status(ErrorMsg(TErrorCode::INTERNAL_ERROR,
Should this be converted to debug action ?
http://gerrit.cloudera.org:8080/#/c/12297/5/be/src/service/control-service.cc@186
PS5, Line 186:
:
Should this be converted to debug action ?
--
To view, visit http://gerrit.cloudera.org:8080/12297
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941
Gerrit-Change-Number: 12297
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Andrew She
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 Notes: Testing is still a WIP. Will trigger a full-suite of tests for both when USE_CDP_HIVE is true and USE_CDP_HIVE is false. Will update the patch as and when the issues are found in the tests. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml 27 files changed, 1,050 insertions(+), 268 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/3 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. Testing: - Ran core tests Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M buildall.sh M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 7 files changed, 76 insertions(+), 218 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/5 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 17:11:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py@508 PS3, Line 508: toolchain_host = os.environ["IMPALA_TOOLCHAIN_HOST"] : cdh_build_number = os.environ["CDH_BUILD_NUMBER"] > nit: looks like it's safe to just use os.environ['IMPALA_TOOLCHAIN_HOST'] a Done http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py@538 PS3, Line 538: if use_cdp_hive: : cdp_components > Since use_cdp_hive is not used again: Ah yeah, this is duplicated in L512. Done. http://gerrit.cloudera.org:8080/#/c/13002/3/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/13002/3/buildall.sh@540 PS3, Line 540: > extraneous #.. There's two more in this parragraph Done -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 16 Apr 2019 17:11:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4027/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 17:11:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13025 to look at the new patch set (#2). Change subject: IMPALA-8415: Fix tests broken by storage layer information .. IMPALA-8415: Fix tests broken by storage layer information Storage layer information was added to the query profile by IMPALA-6050. This broke some tests on exhaustive and s3 runs due to changes in formatting. This fixes the issues: 1. Replace HDFS SCAN with $FILESYSTEM_NAME SCAN in some test files 2. Add $FILESYSTEM_NAME to partition information string Testing: - Ran exhaustive HDFS tests - Ran s3 tests Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 --- M testdata/workloads/functional-query/queries/QueryTest/corrupt-stats.test M testdata/workloads/functional-query/queries/QueryTest/set.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/metadata/test_ddl.py 4 files changed, 36 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13025/2 -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13025/1/tests/metadata/test_ddl.py File tests/metadata/test_ddl.py: http://gerrit.cloudera.org:8080/#/c/13025/1/tests/metadata/test_ddl.py@30 PS1, Line 30: from tests.util.filesystem_utils import WAREHOUSE, IS_HDFS, IS_S3, IS_ADLS, \ > Prefer enclosing list in parens Good point, fixed -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 17:10:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Hector Acosta has posted comments on this change. (
http://gerrit.cloudera.org:8080/13002 )
Change subject: Bump CDP_BUILD_NUMBER to 1013201
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py@508
PS3, Line 508: toolchain_host = os.environ.get("IMPALA_TOOLCHAIN_HOST")
: cdh_build_number = os.environ.get("CDH_BUILD_NUMBER")
nit: looks like it's safe to just use os.environ['IMPALA_TOOLCHAIN_HOST'] and
os.environ['CDH_BUILD_NUMBER']
http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py@538
PS3, Line 538: use_cdp_hive = os.getenv("USE_CDP_HIVE") == "true"
: if use_cdp_hive:
Since use_cdp_hive is not used again:
if os.environ.get('USE_CDP_HIVE') == true:
http://gerrit.cloudera.org:8080/#/c/13002/3/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/13002/3/buildall.sh@540
PS3, Line 540: #
extraneous #.. There's two more in this parragraph
--
To view, visit http://gerrit.cloudera.org:8080/13002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada
Gerrit-Change-Number: 13002
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 16 Apr 2019 16:58:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13025 ) Change subject: IMPALA-8415: Fix tests broken by storage layer information .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13025/1/tests/metadata/test_ddl.py File tests/metadata/test_ddl.py: http://gerrit.cloudera.org:8080/#/c/13025/1/tests/metadata/test_ddl.py@30 PS1, Line 30: from tests.util.filesystem_utils import WAREHOUSE, IS_HDFS, IS_S3, IS_ADLS, \ Prefer enclosing list in parens -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 16:56:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13002 ) Change subject: Bump CDP_BUILD_NUMBER to 1013201 .. Bump CDP_BUILD_NUMBER to 1013201 This patch bumps the CDP_BUILD_NUMBER to 1013201. This patch also refactors the bootstrap_toolchain.py to be more generic for dealing with CDP components, e.g. Ranger and Hive 3. Testing: - Ran core tests Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M buildall.sh M impala-parent/pom.xml M testdata/cluster/ranger/ranger-admin-default-site.xml.template D testdata/cluster/ranger/security-applicationContext.xml 7 files changed, 76 insertions(+), 217 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13002/3 -- To view, visit http://gerrit.cloudera.org:8080/13002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada Gerrit-Change-Number: 13002 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8415: Fix tests broken by storage layer information
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13025 Change subject: IMPALA-8415: Fix tests broken by storage layer information .. IMPALA-8415: Fix tests broken by storage layer information Storage layer information was added to the query profile by IMPALA-6050. This broke some tests on exhaustive and s3 runs due to changes in formatting. This fixes the issues: 1. Replace HDFS SCAN with $FILESYSTEM_NAME SCAN in some test files 2. Add $FILESYSTEM_NAME to partition information string Testing: - Ran exhaustive HDFS tests - Ran s3 tests Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 --- M testdata/workloads/functional-query/queries/QueryTest/corrupt-stats.test M testdata/workloads/functional-query/queries/QueryTest/set.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/metadata/test_ddl.py 4 files changed, 32 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13025/1 -- To view, visit http://gerrit.cloudera.org:8080/13025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I11c6ab9c888464a0f0daaf8a7a6f565d25731872 Gerrit-Change-Number: 13025 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/2 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 21: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/12481/21//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12481/21//COMMIT_MSG@21 PS21, Line 21: casts will fail with an error, just like invalid DECIMAL_V2 casts): It could be added that "while failed casts to other types do no lead to warning or error". -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 21 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 14:04:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 12: Code-Review+1 (5 comments) http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/hdfs-parquet-scanner.cc@618 PS11, Line 618: // If the row group overlaps with the optional: This loop grew really large, > 100 lines, doesn't even fit to my huge monitor. :) At this point it was decided that the row group belongs to the scanner. The rest could be extracted to functions like Status ApplyStatFilters(const parquet::RowGroup& row_group, bool* row_group_skipped) and Status ScanDictioneries(const parquet::RowGroup& row_group, bool* row_group_skipped) http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h File be/src/exec/parquet/parquet-page-index.h: http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h@39 PS11, Line 39: /// It reads the raw bytes of the whole page index and stores it in an : /// internal buffer. : /// It doesn't expect that the Page index in a particular layout, it only : /// expects that the whole page index layed out continuously in the file. : /// It needs to be called before the serialization methods. : Status ReadAll(); Can you mention that this reads the page index for all row groups? For me this was not intuitive. The function's name could also reflect this, as "ReadAll" could also mean reading it for all column chunks in the row group. http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h@61 PS11, Line 61: /// Common helper for deserialization nit: other member comments have . at the end http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h@65 PS11, Line 65: /// The scanner that created this object nit: other member comments have . at the end http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.cc File be/src/exec/parquet/parquet-page-index.cc: http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.cc@81 PS11, Line 81: > 1. The ScanNode creates and deletes scanners for the file splits. This mean Sorry for the wrong assumptions, I got lost somewhere in HdfsParquetScanner::NextRowGroup(). I still think though that we hold onto memory that we do not want to use later. Also, theoretically there can be huge Parquet files with lot of row groups, so the complete page index can be arbitrarily large. Would it be problematic to read the page index for every row group separately during ProcessPageIndex()? This would mean that if the row group is skipped because of row group level stats or because of it is out of the split (the latter is the normal case for multi row group files I guess), then we would avoid reading it's page index. Regardless, for the single row group case, releasing the buffer if we are reading the last row group would be enough. This would avoid the memory wasting for Parquet files written by Impala. I am ok with dealing with this in another patch. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 12 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 16 Apr 2019 14:01:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Log scanner Open() errors
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13018 ) Change subject: Log scanner Open() errors .. Log scanner Open() errors While the status is propagated to the runtime profile, it helps to have it in the log file in cases where the profile is not available. Testing: Verified the message is logged by setting PREPARE_SCANNER debug action. Change-Id: I595d509ca24077b924923fa9aa4cdb912c21c9f2 Reviewed-on: http://gerrit.cloudera.org:8080/13018 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-scan-node.cc 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13018 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I595d509ca24077b924923fa9aa4cdb912c21c9f2 Gerrit-Change-Number: 13018 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Log scanner Open() errors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13018 ) Change subject: Log scanner Open() errors .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13018 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I595d509ca24077b924923fa9aa4cdb912c21c9f2 Gerrit-Change-Number: 13018 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 16 Apr 2019 11:56:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12927 ) Change subject: IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger .. IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger This patch updates the Ranger authorization checker code to deny access when column masking and row filtering is enabled in Ranger for queries that that have columns/tables specified in column mask and row filter policies. This is to prevent data leak, such that the data that is masked/filtered in Hive should not be visible at all in Impala until Impala has full support for column masking and row filtering. Testing: - Added tests in AuthorizationStmtTest to test queries with column masking and row filtering enabled. - Ran all FE tests - Ran all E2E tests Change-Id: If46b4bf24d916e4a4ea8a36ff4acfd95d5f45c8e Reviewed-on: http://gerrit.cloudera.org:8080/12927 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 11 files changed, 407 insertions(+), 49 deletions(-) Approvals: Fredy Wijaya: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If46b4bf24d916e4a4ea8a36ff4acfd95d5f45c8e Gerrit-Change-Number: 12927 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12927 ) Change subject: IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If46b4bf24d916e4a4ea8a36ff4acfd95d5f45c8e Gerrit-Change-Number: 12927 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 16 Apr 2019 09:40:18 + Gerrit-HasComments: No
