[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5051/ -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 04 Oct 2019 03:47:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 ) Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4727/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 04 Oct 2019 03:31:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 ) Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h File be/src/exec/blocking-plan-root-sink.h: http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h@72 PS4, Line 72: bool* eos, const uint64_t timeout_us) override; > I'd prefer non-const for arguments that are passed by value, it just adds n Done http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc@694 PS4, Line 694: max(static_cast(1), > I think there's still a possibility of underflow cause it's unsigned. I thi Done. Changed everything to use int64_t. -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 04 Oct 2019 02:50:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14332 to look at the new patch set (#5). Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only applies to fetch requests against a query that has already transitioned to the 'FINISHED' state. This patch changes the timeout so that it applies to queries in the 'RUNNING' state as well. Before this patch, fetch requests issued while a query was 'RUNNING' blocked until the query transitioned to the 'FINISHED' state, and then it fetched results and returned them. After this patch, fetch requests against queries in the 'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return. For HS2 clients, fetch requests that return while a query is 'RUNNING' set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients, fetch requests that return while a query is 'RUNNING' set the 'ready' flag to false. For both clients, hasMoreRows is set to true. If the following sequence of events occurs: * A fetch request is issued and blocks on a 'RUNNING' query * The query transitions to the 'FINISHED' state * The fetch request attempts to read multiple batches Then the time spent waiting for the query to finish is deducted from the timeout used when waiting for rows to be produced by the Coordinator fragment. Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not being converted properly. Tests: * Moved existing fetch timeout tests from hs2/test_fetch.py into a new test file hs2/test_fetch_timeout.py. * Added several new tests to hs2/test_fetch_timeout.py to validate that the timeout is applied to 'RUNNING' queries and that the timeout applies across a 'RUNNING' and 'FINISHED' query. * Added new tests to query_test/test_fetch.py to validate the timeout while using the Beeswax protocol. Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 --- M be/src/common/status.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/util/error-util.cc M common/thrift/ImpalaService.thrift M tests/hs2/hs2_test_suite.py M tests/hs2/test_fetch.py A tests/hs2/test_fetch_timeout.py M tests/query_test/test_fetch.py 19 files changed, 562 insertions(+), 188 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/5 -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4726/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 04 Oct 2019 01:14:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 04 Oct 2019 01:10:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests Disable the tests TestResultSpooling::test_full_queue(_large_fetch) until we figure out why they are flaky. Replace the sleep in TestAdmissionController::test_release_backend with assert_eventually to reduce flakiness. Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Reviewed-on: http://gerrit.cloudera.org:8080/14337 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_admission_controller.py M tests/query_test/test_result_spooling.py 2 files changed, 6 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar
[native-toolchain-CR] IMPALA-8978: Fix TLS issues in Centos 6 docker image build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14366 ) Change subject: IMPALA-8978: Fix TLS issues in Centos 6 docker image build .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I938a949a16eaf3291e765d67fbb1fd8912b8424f Gerrit-Change-Number: 14366 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 04 Oct 2019 00:36:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14357 )
Change subject: IMPALA-8998: admission control accounting for mt_dop
..
Patch Set 11: Code-Review+1
(5 comments)
Carry bikram's +1
http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:
http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/scheduler.cc@689
PS8, Line 689: for (auto& backend : per_backend_params) {
: int be_max_instances = 0;
: // Instances for a fragment are clustered together because
of how the vector is
: // constructed above. So we can compute the max # of
instances of any fragment
: // with a single pass over the vector.
: const FragmentExecParams* curr_fragment = nullptr;
: int curr_instance_count = 0; // Number of instances of the
current fragment seen.
: for (auto& finstance : backend.second.instance_params) {
: if (curr_fragment == nullptr ||
: curr_fragment != &finstance->fragment_exec_params) {
: curr_fragment = &finstance->fragment_exec_params;
: curr_instance_count = 0;
: }
: ++curr_instance_count;
: be_max_instances = max(be_max_instances,
curr_instance_count);
: }
: backend.second.slots_to_use = be_max_instances;
: }
> nit: seems like we are iterating over per_backend_params a few times after
I'd prefer multiple passes that do one thing each rather than a single pass
that does multiple things. I don't think the loop overhead should be
significant.
http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_admission_controller.py
File tests/custom_cluster/test_admission_controller.py:
http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_admission_controller.py@882
PS8, Line 882: Test that queue details appear in the profile when queued based
on num_queries
> nit: update description
Done
http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_admission_controller.py@903
PS8, Line 903: num_queries
> nit: update message
Done
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/metric_verifier.py
File tests/verifiers/metric_verifier.py:
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/metric_verifier.py@69
PS11, Line 69: wait_for_backend_admission_control_state
> nit: (feel free to ignore) how about "wait_for_idle_backend_state"?
I think I'll stick with this name, even though it's a little awkward. I kinda
wanted capture that it wasn't the state of the actual backends, but rather the
admission control state.
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/test_verify_metrics.py
File tests/verifiers/test_verify_metrics.py:
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/test_verify_metrics.py@48
PS11, Line 48: test_verify_backends
> nit: (feel free to ignore) how about "test_backends_are_idle"
Done
--
To view, visit http://gerrit.cloudera.org:8080/14357
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5
Gerrit-Change-Number: 14357
Gerrit-PatchSet: 11
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 04 Oct 2019 00:32:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#12). Change subject: IMPALA-8998: admission control accounting for mt_dop .. IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. --admission_control_slots is added as a new flag that replaces --max_concurrent_queries, since the name better reflects the concept. --max_concurrent_queries is kept for backwards compatibility and has the same meaning as --admission_control_slots. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * End-to-end admission test that exercises the admit immediately and queueing code paths. Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/util/auto_scaler.py M tests/verifiers/metric_verifier.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 19 files changed, 410 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/12 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] IMPALA-8978: Fix TLS issues in Centos 6 docker image build
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14366 Change subject: IMPALA-8978: Fix TLS issues in Centos 6 docker image build .. IMPALA-8978: Fix TLS issues in Centos 6 docker image build Everything needs TLSv1.2 now, and the version of nss that ships with Centos 6.6 is too old. We already knew this and were installing the nss from Centos 6.8, but we were installing it after all the yum-install commands. The yum-install commands now fail with TLS issues, so this installs the new nss before invoking any other yum-install commands. Testing: - Built the redhat6 docker image Change-Id: I938a949a16eaf3291e765d67fbb1fd8912b8424f --- M docker/redhat6.df 1 file changed, 5 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/66/14366/1 -- To view, visit http://gerrit.cloudera.org:8080/14366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I938a949a16eaf3291e765d67fbb1fd8912b8424f Gerrit-Change-Number: 14366 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4725/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:50:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4724/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:48:33 + Gerrit-HasComments: No
[native-toolchain-CR] Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14361 ) Change subject: Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14 .. Patch Set 1: Verified+1 Ubuntu 14.04 toolchain build succeeded. Merging. -- To view, visit http://gerrit.cloudera.org:8080/14361 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I719c3ac58d3087042e060e3668e12d24720c1d68 Gerrit-Change-Number: 14361 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:45:37 + Gerrit-HasComments: No
[native-toolchain-CR] Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14361 ) Change subject: Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14 .. Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14 Kudu no longer compiles on Java 7, and these platforms are getting very old. This removes the Ubuntu 12 and Debian 7 docker files. It changes Ubuntu 14 to use Java 8 (which it gets by adding an extra apt repository). Removing Debian 7 makes IMPALA-8992 moot. Testing: - Ran Ubuntu1404 docker build Change-Id: I719c3ac58d3087042e060e3668e12d24720c1d68 Reviewed-on: http://gerrit.cloudera.org:8080/14361 Reviewed-by: Tim Armstrong Tested-by: Joe McDonnell --- D docker/debian/7/sources.list D docker/debian7.df D docker/ubuntu1204.df M docker/ubuntu1404.df 4 files changed, 4 insertions(+), 79 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/14361 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I719c3ac58d3087042e060e3668e12d24720c1d68 Gerrit-Change-Number: 14361 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/14357 )
Change subject: IMPALA-8998: admission control accounting for mt_dop
..
Patch Set 11: Code-Review+1
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:
http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/scheduler.cc@689
PS8, Line 689: for (auto& backend : per_backend_params) {
: int be_max_instances = 0;
: // Instances for a fragment are clustered together because
of how the vector is
: // constructed above. So we can compute the max # of
instances of any fragment
: // with a single pass over the vector.
: const FragmentExecParams* curr_fragment = nullptr;
: int curr_instance_count = 0; // Number of instances of the
current fragment seen.
: for (auto& finstance : backend.second.instance_params) {
: if (curr_fragment == nullptr ||
: curr_fragment != &finstance->fragment_exec_params) {
: curr_fragment = &finstance->fragment_exec_params;
: curr_instance_count = 0;
: }
: ++curr_instance_count;
: be_max_instances = max(be_max_instances,
curr_instance_count);
: }
: backend.second.slots_to_use = be_max_instances;
: }
nit: seems like we are iterating over per_backend_params a few times after its
construction. Do you think we can merge them or would that impact readability?
http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_admission_controller.py
File tests/custom_cluster/test_admission_controller.py:
http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_admission_controller.py@882
PS8, Line 882: Test that queue details appear in the profile when queued based
on num_queries
nit: update description
http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_admission_controller.py@903
PS8, Line 903: num_queries
nit: update message
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/metric_verifier.py
File tests/verifiers/metric_verifier.py:
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/metric_verifier.py@69
PS11, Line 69: wait_for_backend_admission_control_state
nit: (feel free to ignore) how about "wait_for_idle_backend_state"?
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/test_verify_metrics.py
File tests/verifiers/test_verify_metrics.py:
http://gerrit.cloudera.org:8080/#/c/14357/11/tests/verifiers/test_verify_metrics.py@48
PS11, Line 48: test_verify_backends
nit: (feel free to ignore) how about "test_backends_are_idle"
--
To view, visit http://gerrit.cloudera.org:8080/14357
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5
Gerrit-Change-Number: 14357
Gerrit-PatchSet: 11
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 03 Oct 2019 23:43:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8826: [DOCS] Add docs for PLAN ROOT SINK and result spooling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14314 ) Change subject: IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling .. Patch Set 5: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/506/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe Gerrit-Change-Number: 14314 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:30:28 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Copy edited to remove a dummy paragraph
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14364 ) Change subject: [DOCS] Copy edited to remove a dummy paragraph .. [DOCS] Copy edited to remove a dummy paragraph Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 Reviewed-on: http://gerrit.cloudera.org:8080/14364 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_parquet_object_store_split_size.xml 1 file changed, 0 insertions(+), 1 deletion(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14364 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 Gerrit-Change-Number: 14364 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9005: distcc server bootstrapping improvements
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14363 ) Change subject: IMPALA-9005: distcc server bootstrapping improvements .. Patch Set 1: (4 comments) Thanks for the review. This is kinda raw and might not handle a bunch of edge cases, but wanted to get it checked in at least instead of sitting around on my own system. http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh File bin/distcc/distcc_server_bootstrap.sh: http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@26 PS1, Line 26: Arguments are forwarded to : # distcc_server_setup.sh to specify IP ranges, etc. > Nit: Are we ok with the defaults for CCACHE_DIR/CCACHE_SIZE that are set in Added a mention. I didn't want to re-document those options so I made the explanation generic. http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@43 PS1, Line 43: fi > Will add an else that fails the script. Done http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@62 PS1, Line 62: # Set HOME as workaround for ccache trying to access /.ccache : HOME=$(pwd) > To check my understanding: this is ccache usage from these setup commands a Done. Updated the comment to explain better http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@70 PS1, Line 70: . bin/impala-config.sh : ./infra/python/deps/download_requirements : DOWNLOAD_CDH_COMPONENTS=false ./bin/bootstrap_toolchain.py > Super-nit: indentation here is a single space, but it should be two spaces Done -- To view, visit http://gerrit.cloudera.org:8080/14363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308 Gerrit-Change-Number: 14363 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:23:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Copy edited to remove a dummy paragraph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14364 ) Change subject: [DOCS] Copy edited to remove a dummy paragraph .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/505/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14364 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 Gerrit-Change-Number: 14364 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 23:29:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8826: [DOCS] Add docs for PLAN ROOT SINK and result spooling
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14314 ) Change subject: IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe Gerrit-Change-Number: 14314 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:28:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Code-Review-2 Tests are still failing, running them again to see if they are flaky (but will not submit until we fix them). Marking -2 to make sure we don't accidentally submit the change. -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:25:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5051/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:26:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8826: [DOCS] Add docs for PLAN ROOT SINK and result spooling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14314 ) Change subject: IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling .. Patch Set 5: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/506/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe Gerrit-Change-Number: 14314 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:13:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1071: Distributable python package for impala-shell
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14181 ) Change subject: IMPALA-1071: Distributable python package for impala-shell .. Patch Set 6: Code-Review+2 I mentioned to David that there seemed to be another issue with exception handling and "TSocket read 0 bytes" causing the shell to exit after the impala service was restarted. But that doesn't necessarily have to block this going in. -- To view, visit http://gerrit.cloudera.org:8080/14181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib8c745bf6a16f0c039430152745a2f00e044 Gerrit-Change-Number: 14181 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Dinesh Garg (430) Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:14:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#9). Change subject: IMPALA-8998: admission control accounting for mt_dop .. IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. --admission_control_slots is added as a new flag that replaces --max_concurrent_queries, since the name better reflects the concept. --max_concurrent_queries is kept for backwards compatibility and has the same meaning as --admission_control_slots. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * End-to-end admission test that exercises the admit immediately and queueing code paths. Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/util/auto_scaler.py M tests/verifiers/metric_verifier.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 19 files changed, 406 insertions(+), 83 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/9 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9005: distcc server bootstrapping improvements
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14363 ) Change subject: IMPALA-9005: distcc server bootstrapping improvements .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh File bin/distcc/distcc_server_bootstrap.sh: http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@43 PS1, Line 43: fi Will add an else that fails the script. -- To view, visit http://gerrit.cloudera.org:8080/14363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308 Gerrit-Change-Number: 14363 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 22:58:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8826: [DOCS] Add docs for PLAN ROOT SINK and result spooling
Hello Michael Ho, Sahil Takiar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14314 to look at the new patch set (#5). Change subject: IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling .. IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe --- M docs/impala.ditamap M docs/impala_keydefs.ditamap M docs/shared/ImpalaVariables.xml M docs/topics/impala_client.xml A docs/topics/impala_fetch_rows_timeout_ms.xml A docs/topics/impala_max_result_spooling_mem.xml A docs/topics/impala_max_spilled_result_spooling_mem.xml A docs/topics/impala_query_results_spooling.xml A docs/topics/impala_spool_query_results.xml 9 files changed, 405 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/14314/5 -- To view, visit http://gerrit.cloudera.org:8080/14314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe Gerrit-Change-Number: 14314 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8826: [DOCS] Add docs for PLAN ROOT SINK and result spooling
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14314 ) Change subject: IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14314/4/docs/topics/impala_spool_query_results.xml File docs/topics/impala_spool_query_results.xml: http://gerrit.cloudera.org:8080/#/c/14314/4/docs/topics/impala_spool_query_results.xml@49 PS4, Line 49: on a persistent storage, > delete this, memory is not persistent Done -- To view, visit http://gerrit.cloudera.org:8080/14314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe Gerrit-Change-Number: 14314 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:12:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Copy edited to remove a dummy paragraph
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14364 ) Change subject: [DOCS] Copy edited to remove a dummy paragraph .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14364 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 Gerrit-Change-Number: 14364 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 23:05:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#11). Change subject: IMPALA-8998: admission control accounting for mt_dop .. IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. --admission_control_slots is added as a new flag that replaces --max_concurrent_queries, since the name better reflects the concept. --max_concurrent_queries is kept for backwards compatibility and has the same meaning as --admission_control_slots. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * End-to-end admission test that exercises the admit immediately and queueing code paths. Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/util/auto_scaler.py M tests/verifiers/metric_verifier.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 19 files changed, 409 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/11 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9005: distcc server bootstrapping improvements
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14363 ) Change subject: IMPALA-9005: distcc server bootstrapping improvements .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh File bin/distcc/distcc_server_bootstrap.sh: http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@26 PS1, Line 26: Arguments are forwarded to : # distcc_server_setup.sh to specify IP ranges, etc. Nit: Are we ok with the defaults for CCACHE_DIR/CCACHE_SIZE that are set in distcc_server_setup.sh? Is this something we should mention as a configurable parameter? http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@62 PS1, Line 62: # Set HOME as workaround for ccache trying to access /.ccache : HOME=$(pwd) To check my understanding: this is ccache usage from these setup commands and not really important for later. It looks like bootstrap_virtualenv.py invokes ccache. If this is right, can you add a comment saying that this only applies to these setup commands? Nit: if we are setting HOME directly, then maybe we don't need the -H on the sudo command? http://gerrit.cloudera.org:8080/#/c/14363/1/bin/distcc/distcc_server_bootstrap.sh@70 PS1, Line 70: . bin/impala-config.sh : ./infra/python/deps/download_requirements : DOWNLOAD_CDH_COMPONENTS=false ./bin/bootstrap_toolchain.py Super-nit: indentation here is a single space, but it should be two spaces -- To view, visit http://gerrit.cloudera.org:8080/14363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308 Gerrit-Change-Number: 14363 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:07:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 8: (7 comments) I won't carry +1 since the startup flag change was kinda significant. http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.h File be/src/runtime/exec-env.h: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.h@267 PS8, Line 267: /// The maximum number of admission slots that should be used on this host. > Maybe explain (here or elsewhere) what "slot" means, e.g. that it's concept I added a bit more of an explanation here. I don't want to get too in depth in this comment, but agree some more context is useful. Tried to make it roughly consistent with admit_mem_limit as far as the detail level. http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.cc@90 PS8, Line 90: "(Advanced) The maximum degree of parallelism to run queries with on this backend. " > Maybe we should create a new flag with a better name and the same behavior, That's a good point. I just added a new flag that takes precedence over the old one. We can probably leave the old one around indefinitely TBH, the code to do the fallback isn't complex. http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.h@252 PS8, Line 252: > nit: double space Done http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.cc@563 PS8, Line 563: HasAvailableSlot > nit: rename to HasAvailableSlots()? Done http://gerrit.cloudera.org:8080/#/c/14357/8/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test: http://gerrit.cloudera.org:8080/#/c/14357/8/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test@3 PS8, Line 3: 4 > This is because we set mt_dop=4 and 24/3 > 4, right? It might be good to me Done http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_executor_groups.py@218 PS8, Line 218: gets > nit: get Done http://gerrit.cloudera.org:8080/#/c/14357/8/tests/verifiers/metric_verifier.py File tests/verifiers/metric_verifier.py: http://gerrit.cloudera.org:8080/#/c/14357/8/tests/verifiers/metric_verifier.py@69 PS8, Line 69: wait_for_backends_state > maybe this method could have a more descriptive name? Done -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:07:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/14357/9/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/14357/9/tests/custom_cluster/test_executor_groups.py@57 PS9, Line 57: t flake8: E501 line too long (91 > 90 characters) http://gerrit.cloudera.org:8080/#/c/14357/9/tests/custom_cluster/test_executor_groups.py@67 PS9, Line 67: ] flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 23:07:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Copy edited to remove a dummy paragraph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14364 ) Change subject: [DOCS] Copy edited to remove a dummy paragraph .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/505/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14364 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 Gerrit-Change-Number: 14364 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 23:05:40 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Copy edited to remove a dummy paragraph
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14364 Change subject: [DOCS] Copy edited to remove a dummy paragraph .. [DOCS] Copy edited to remove a dummy paragraph Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 --- M docs/topics/impala_parquet_object_store_split_size.xml 1 file changed, 0 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/14364/1 -- To view, visit http://gerrit.cloudera.org:8080/14364 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2a12ada7d5b08fabf25cf3df96793053dde08af3 Gerrit-Change-Number: 14364 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8830: Fix executor group assignment of coordinator only queries
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14183 ) Change subject: IMPALA-8830: Fix executor group assignment of coordinator only queries .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/admission-controller.h@973 PS5, Line 973: /// the query is a coordinator only query, then a reserved empty group is returned nit: punctuation http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/scheduler.h File be/src/scheduling/scheduler.h: http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/scheduler.h@72 PS5, Line 72: /// ranges in the query exec request. Mention here that executor_config must contain a non-empty group unless IsCoordinatorOnlyQuery() is true? http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/scheduler.cc@143 PS5, Line 143: bool exec_at_coord = (fragment.partition.type == TPartitionType::UNPARTITIONED); Can we add a DCHECK to verify that exec_at_coord is true if executor_config only contains the empty group? It's probably implicitly checked by the assignment computation, but I think it would be good to tie this together with IsCoordinatorQuery(). http://gerrit.cloudera.org:8080/#/c/14183/5/be/src/scheduling/scheduler.cc@466 PS5, Line 466: // case. I think here it would help future-us if we include the purpose of this group, e.g similar to this: > This group is used to schedule fragments where the caller of this method has > determined that they need to be scheduled on the coordinator only. Note that > this includes queries where the whole query should run on the coordinator, as > is determined by Scheduler::IsCoordinatorOnlyQuery(). For those queries, the > AdmissionController will pass an empty executor group and rely on this method > being called with exec_at_coord = true. -- To view, visit http://gerrit.cloudera.org:8080/14183 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe098032744aa20bbbe4faddfc67e7a46ce03d5 Gerrit-Change-Number: 14183 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 03 Oct 2019 23:03:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9005: distcc server bootstrapping improvements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14363 ) Change subject: IMPALA-9005: distcc server bootstrapping improvements .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4723/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308 Gerrit-Change-Number: 14363 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 03 Oct 2019 22:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5049/ -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 22:32:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14338 ) Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. IMPALA-8971: [DOCS] Documented the new query option for Parquet split size Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Reviewed-on: http://gerrit.cloudera.org:8080/14338 Reviewed-by: Sahil Takiar Tested-by: Impala Public Jenkins --- M docs/impala.ditamap M docs/topics/impala_parquet.xml A docs/topics/impala_parquet_object_store_split_size.xml M docs/topics/impala_s3.xml 4 files changed, 68 insertions(+), 45 deletions(-) Approvals: Sahil Takiar: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14338 ) Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/504/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 22:37:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14338 ) Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 22:23:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9005: distcc server bootstrapping improvements
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14363 Change subject: IMPALA-9005: distcc server bootstrapping improvements .. IMPALA-9005: distcc server bootstrapping improvements Adds a script distcc_server_bootstrap.sh that can be run in a standalone way on an ubuntu 16.04 server. A next step could be to build a docker image that can be deployed as an impala distcc server. Testing: Ran in an ubuntu 16.04 docker image and then confirmed I could add it to my BUILD_FARM and compile things with it. Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308 --- A bin/distcc/distcc_server_bootstrap.sh M bin/distcc/distcc_server_setup.sh 2 files changed, 83 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/14363/1 -- To view, visit http://gerrit.cloudera.org:8080/14363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9e382e90f68a5a2fe2917184fa4603bed492a308 Gerrit-Change-Number: 14363 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8826: [DOCS] Add docs for PLAN ROOT SINK and result spooling
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14314 ) Change subject: IMPALA-8826: [DOCS] Add docs for PLAN_ROOT_SINK and result spooling .. Patch Set 4: Code-Review+2 (1 comment) Minor comments, otherwise LGTM. http://gerrit.cloudera.org:8080/#/c/14314/4/docs/topics/impala_spool_query_results.xml File docs/topics/impala_spool_query_results.xml: http://gerrit.cloudera.org:8080/#/c/14314/4/docs/topics/impala_spool_query_results.xml@49 PS4, Line 49: on a persistent storage, delete this, memory is not persistent -- To view, visit http://gerrit.cloudera.org:8080/14314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78bfceb225d25078c54c1ed8f88ca250ef42dafe Gerrit-Change-Number: 14314 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 22:15:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14356 ) Change subject: IMPALA-8587: show grant does not produce correct privileges .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4722/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65 Gerrit-Change-Number: 14356 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 22:15:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14338 ) Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/504/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 22:12:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Hello David Rorke, Sahil Takiar, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14338 to look at the new patch set (#2). Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. IMPALA-8971: [DOCS] Documented the new query option for Parquet split size Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a --- M docs/impala.ditamap M docs/topics/impala_parquet.xml A docs/topics/impala_parquet_object_store_split_size.xml M docs/topics/impala_s3.xml 4 files changed, 68 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/14338/2 -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14338 ) Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14338/1/docs/topics/impala_parquet_object_store_split_size.xml File docs/topics/impala_parquet_object_store_split_size.xml: http://gerrit.cloudera.org:8080/#/c/14338/1/docs/topics/impala_parquet_object_store_split_size.xml@41 PS1, Line 41: bigger than > greater than or equal to Done -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 22:12:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14338 ) Change subject: IMPALA-8971: [DOCS] Documented the new query option for Parquet split size .. Patch Set 1: Code-Review+2 (1 comment) One minor comment, otherwise LGTM. http://gerrit.cloudera.org:8080/#/c/14338/1/docs/topics/impala_parquet_object_store_split_size.xml File docs/topics/impala_parquet_object_store_split_size.xml: http://gerrit.cloudera.org:8080/#/c/14338/1/docs/topics/impala_parquet_object_store_split_size.xml@41 PS1, Line 41: bigger than greater than or equal to -- To view, visit http://gerrit.cloudera.org:8080/14338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a Gerrit-Change-Number: 14338 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 22:09:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4721/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 21:58:19 + Gerrit-HasComments: No
[native-toolchain-CR] Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14361 ) Change subject: Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14361 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I719c3ac58d3087042e060e3668e12d24720c1d68 Gerrit-Change-Number: 14361 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 21:49:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Fang-Yu Rao has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/14356 )
Change subject: IMPALA-8587: show grant does not produce correct privileges
..
IMPALA-8587: show grant does not produce correct privileges
Currently the show grant command cannot produce correct privileges when the
list of resources in a RangerPolicy contains '*', a wildcard denoting all the
possible resources.
For example, when a user 'admin' having 'all' as one of its allowed operations
in a Ranger policy, if the field of 'database' in this RangerPolicy contains
only '*', given a SQL statement 'show grant user admin on database functional',
the current logic in RangerImpaladAuthorizationManager.java would consider that
the user 'admin' does not have any privilege on the database 'functional',
whereas in fact the user 'admin' should have all privileges for each database
on this server.
This patch resolves the issue by taking into consideration the case in which
the field of resources could contain a wildcard when determining whether or not
the name of the resource of interest ('functional' in the example above) is
contained in the list of resources the user ('admin' in the example above) has
privileges on.
Testing:
Tested this patch against the updated test_ranger.py.
Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65
---
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M tests/authorization/test_ranger.py
2 files changed, 67 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/14356/3
--
To view, visit http://gerrit.cloudera.org:8080/14356
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65
Gerrit-Change-Number: 14356
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8996: fix test show create table in test zorder.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14359 ) Change subject: IMPALA-8996: fix test_show_create_table in test_zorder.py .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic316224325eec64d9b1e570854a74c0372084a4a Gerrit-Change-Number: 14359 Gerrit-PatchSet: 2 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 03 Oct 2019 21:21:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8996: fix test show create table in test zorder.py
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14359 ) Change subject: IMPALA-8996: fix test_show_create_table in test_zorder.py .. IMPALA-8996: fix test_show_create_table in test_zorder.py IMPALA-8593 extended the filtered tbl properties in test_show_create_table.py. This change was not propagated to test_zorder.py which created a duplicate of the former in order to run its tests in a custom cluster environment. Removed the "show create table" tests from test_zorder.py. These tests are not critical, but can cause more issues like this. Also, Z-order tests are in a separate file only until custom cluster is required. After merging IMPALA-8755, and removing the feature flag (which is the reason for the custom cluster), the tests can be merged into show-create-table.test (from show-create-table-zorder.test). Change-Id: Ic316224325eec64d9b1e570854a74c0372084a4a Reviewed-on: http://gerrit.cloudera.org:8080/14359 Reviewed-by: Csaba Ringhofer Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_zorder.py 1 file changed, 1 insertion(+), 218 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic316224325eec64d9b1e570854a74c0372084a4a Gerrit-Change-Number: 14359 Gerrit-PatchSet: 3 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 21:19:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5050/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 21:19:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 4: Code-Review+2 Bumped timeout to 60s. Carrying +2. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 21:18:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14337 to look at the new patch set (#4). Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests Disable the tests TestResultSpooling::test_full_queue(_large_fetch) until we figure out why they are flaky. Replace the sleep in TestAdmissionController::test_release_backend with assert_eventually to reduce flakiness. Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b --- M tests/custom_cluster/test_admission_controller.py M tests/query_test/test_result_spooling.py 2 files changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14337/4 -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-7368: Change supported year range for DATE values to 1..9999
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14349 ) Change subject: IMPALA-7368: Change supported year range for DATE values to 1.. .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5047/ -- To view, visit http://gerrit.cloudera.org:8080/14349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iefdf1c036834763f52d44d0c39a25a1f04e41e07 Gerrit-Change-Number: 14349 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 21:17:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 8: Code-Review+1 (7 comments) LGTM, only had a bunch of nits http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.h File be/src/runtime/exec-env.h: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.h@267 PS8, Line 267: /// The maximum number of admission slots that should be used on this host. Maybe explain (here or elsewhere) what "slot" means, e.g. that it's conceptually equivalent with the number of threads or cpu cores? http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/runtime/exec-env.cc@90 PS8, Line 90: "(Advanced) The maximum degree of parallelism to run queries with on this backend. " Maybe we should create a new flag with a better name and the same behavior, and then deprecate this one on the next major release. With the slot concept, its name is somewhat awkward. If we already mark it as deprecated in 3.4, hopefully not many people would start using it. http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.h@252 PS8, Line 252: nit: double space http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/14357/8/be/src/scheduling/admission-controller.cc@563 PS8, Line 563: HasAvailableSlot nit: rename to HasAvailableSlots()? http://gerrit.cloudera.org:8080/#/c/14357/8/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test: http://gerrit.cloudera.org:8080/#/c/14357/8/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test@3 PS8, Line 3: 4 This is because we set mt_dop=4 and 24/3 > 4, right? It might be good to mention that mt_dop is 4 in the comment. http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/14357/8/tests/custom_cluster/test_executor_groups.py@218 PS8, Line 218: gets nit: get http://gerrit.cloudera.org:8080/#/c/14357/8/tests/verifiers/metric_verifier.py File tests/verifiers/metric_verifier.py: http://gerrit.cloudera.org:8080/#/c/14357/8/tests/verifiers/metric_verifier.py@69 PS8, Line 69: wait_for_backends_state maybe this method could have a more descriptive name? -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 21:11:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5046/ -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 20:51:57 + Gerrit-HasComments: No
[native-toolchain-CR] Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14361 Change subject: Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14 .. Remove Ubuntu 12, Debian 7 and use Java 8 for Ubuntu 14 Kudu no longer compiles on Java 7, and these platforms are getting very old. This removes the Ubuntu 12 and Debian 7 docker files. It changes Ubuntu 14 to use Java 8 (which it gets by adding an extra apt repository). Removing Debian 7 makes IMPALA-8992 moot. Testing: - Ran Ubuntu1404 docker build Change-Id: I719c3ac58d3087042e060e3668e12d24720c1d68 --- D docker/debian/7/sources.list D docker/debian7.df D docker/ubuntu1204.df M docker/ubuntu1404.df 4 files changed, 4 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/61/14361/1 -- To view, visit http://gerrit.cloudera.org:8080/14361 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I719c3ac58d3087042e060e3668e12d24720c1d68 Gerrit-Change-Number: 14361 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 18:43:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. IMPALA-8988: [DOCS] DATE type is supported AVRO tables Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Reviewed-on: http://gerrit.cloudera.org:8080/14340 Tested-by: Impala Public Jenkins Reviewed-by: Alex Rodoni --- M docs/topics/impala_avro.xml M docs/topics/impala_date.xml 2 files changed, 102 insertions(+), 62 deletions(-) Approvals: Impala Public Jenkins: Verified Alex Rodoni: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/503/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 18:40:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4720/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 18:34:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4718/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 18:05:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 3: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py@1337 PS1, Line 1337: # Once the 'lineitem' scan completes, NumCompletedBac > Yeah, maybe discussing in person is best, will find you tomorrow. Fair enough. I suppose all functions waiting for certain condition to become true has a timeout by default so there is no getting away from using some sort of timeout to avoid the test from hanging forever. May be safer to bump the timeout to a higher value such as 60s. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 18:14:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 3: A typo fix -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 18:05:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/503/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 18:05:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Hello Attila Jeges, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14340 to look at the new patch set (#3). Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. IMPALA-8988: [DOCS] DATE type is supported AVRO tables Change-Id: I95f37accddadcba436676498d5cbb34cda281846 --- M docs/topics/impala_avro.xml M docs/topics/impala_date.xml 2 files changed, 102 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/14340/3 -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4719/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:59:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4717/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:32:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/14059/14/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/14059/14/tests/shell/test_shell_interactive.py@335 PS14, Line 335: TMP_DB, TMP_TBL)) > This one is already inside the finally. Would you prefer to wrap it in an a Ah, missed that. -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:41:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5049/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:29:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 14: (11 comments) Addressed all comments in PS16 http://gerrit.cloudera.org:8080/#/c/14059/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java File fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java: http://gerrit.cloudera.org:8080/#/c/14059/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java@175 PS8, Line 175: RunShellCommand.Run(command, /* shouldSucceed */ false, "", > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/14059/14/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/14059/14/tests/common/impala_connection.py@323 PS14, Line 323: if 'NoneType' not in str(e): > Can we restrict this to HTTP connections only? Would be nice not to swallow Done http://gerrit.cloudera.org:8080/#/c/14059/8/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/14059/8/tests/common/impala_test_suite.py@237 PS8, Line 237: h > flake8: E713 test for membership should be 'not in' Done http://gerrit.cloudera.org:8080/#/c/14059/14/tests/common/test_dimensions.py File tests/common/test_dimensions.py: http://gerrit.cloudera.org:8080/#/c/14059/14/tests/common/test_dimensions.py@117 PS14, Line 117: # IMPALA-8864: Older python versions do not support SSLContext object that the thrift http client > nit: long line Done http://gerrit.cloudera.org:8080/#/c/14059/8/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/14059/8/tests/shell/test_shell_interactive.py@28 PS8, Line 28: from time import sleep > flake8: F401 'contextlib.contextmanager' imported but unused Done http://gerrit.cloudera.org:8080/#/c/14059/14/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/14059/14/tests/shell/test_shell_interactive.py@293 PS14, Line 293: self.create_impala_clients() > Should this be in a finally? So that a failure here doesn't cause follow-on Done http://gerrit.cloudera.org:8080/#/c/14059/14/tests/shell/test_shell_interactive.py@335 PS14, Line 335: self.create_impala_clients() > Same here (I think I created this problem in the first place by closing the This one is already inside the finally. Would you prefer to wrap it in an additional level to make it robust against failures in the run_shell commands before? http://gerrit.cloudera.org:8080/#/c/14059/8/tests/util/run_impyla_http_query.py File tests/util/run_impyla_http_query.py: http://gerrit.cloudera.org:8080/#/c/14059/8/tests/util/run_impyla_http_query.py@30 PS8, Line 30: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/14059/8/tests/util/run_impyla_http_query.py@33 PS8, Line 33: I > flake8: F841 local variable 'url' is assigned to but never used Done http://gerrit.cloudera.org:8080/#/c/14059/8/tests/util/run_impyla_http_query.py@46 PS8, Line 46: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/14059/8/tests/util/run_impyla_http_query.py@57 PS8, Line 57: > flake8: E305 expected 2 blank lines after class or function definition, fou Done -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 14 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:28:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14059 to look at the new patch set (#16). Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. IMPALA-8863: Add support to run tests over HTTP/HS2 This change adds support to run backend tests over HTTP using a new version of Impyla. It also adds a test that exercises authentication over HTTP. Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 --- M bin/start-impala-cluster.py M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java A fe/src/test/java/org/apache/impala/customcluster/LdapImpylaHttpTest.java A fe/src/test/java/org/apache/impala/customcluster/RunShellCommand.java M infra/python/deps/compiled-requirements.txt M tests/common/impala_connection.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_hs2.py M tests/custom_cluster/test_session_expiration.py M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/query_test/test_cast_with_format.py M tests/query_test/test_chars.py M tests/query_test/test_date_queries.py M tests/query_test/test_decimal_queries.py M tests/query_test/test_queries.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py A tests/util/run_impyla_http_query.py 21 files changed, 408 insertions(+), 148 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/14059/16 -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 ) Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h File be/src/exec/blocking-plan-root-sink.h: http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h@72 PS4, Line 72: bool* eos, const uint64_t timeout_us) override; I'd prefer non-const for arguments that are passed by value, it just adds noise. http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc@694 PS4, Line 694: max(static_cast(1), I think there's still a possibility of underflow cause it's unsigned. I think this would all be simpler if we used int64_t for the timeouts throughout, since we don't need the extra range of the unsigned type. https://google.github.io/styleguide/cppguide.html#Integer_Types addresses this issue and says to use signed types in this situation. We could address the underflow locally, but the choice of signed vs unsigned is kinda viral because anywhere you convert to signed you have to deal with overflow. -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:27:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#8). Change subject: IMPALA-8998: admission control accounting for mt_dop .. IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * End-to-end admission test that exercises the admit immediately and queueing code paths. Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/verifiers/metric_verifier.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 18 files changed, 360 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/8 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4716/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:15:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/14357/7/tests/verifiers/metric_verifier.py File tests/verifiers/metric_verifier.py: http://gerrit.cloudera.org:8080/#/c/14357/7/tests/verifiers/metric_verifier.py@72 PS7, Line 72: L flake8: F821 undefined name 'LOG' -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:13:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 7: I think this is ready for review. -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 17:13:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#7). Change subject: IMPALA-8998: admission control accounting for mt_dop .. IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * End-to-end admission test that exercises the admit immediately and queueing code paths. Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/verifiers/metric_verifier.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 18 files changed, 361 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/7 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/14357/6/tests/verifiers/metric_verifier.py File tests/verifiers/metric_verifier.py: http://gerrit.cloudera.org:8080/#/c/14357/6/tests/verifiers/metric_verifier.py@72 PS6, Line 72: L flake8: F821 undefined name 'LOG' -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 03 Oct 2019 17:12:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#6). Change subject: IMPALA-8998: admission control accounting for mt_dop .. IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * End-to-end admission test that exercises the admit immediately and queueing code paths. Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/verifiers/metric_verifier.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 18 files changed, 361 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/6 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7368: Change supported year range for DATE values to 1..9999
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14349 ) Change subject: IMPALA-7368: Change supported year range for DATE values to 1.. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5047/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iefdf1c036834763f52d44d0c39a25a1f04e41e07 Gerrit-Change-Number: 14349 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 16:55:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8996: fix test show create table in test zorder.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14359 ) Change subject: IMPALA-8996: fix test_show_create_table in test_zorder.py .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5048/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic316224325eec64d9b1e570854a74c0372084a4a Gerrit-Change-Number: 14359 Gerrit-PatchSet: 2 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 03 Oct 2019 17:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8996: fix test show create table in test zorder.py
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14359 ) Change subject: IMPALA-8996: fix test_show_create_table in test_zorder.py .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic316224325eec64d9b1e570854a74c0372084a4a Gerrit-Change-Number: 14359 Gerrit-PatchSet: 2 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 03 Oct 2019 17:02:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Change supported year range for DATE values to 1..9999
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14349 ) Change subject: IMPALA-7368: Change supported year range for DATE values to 1.. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iefdf1c036834763f52d44d0c39a25a1f04e41e07 Gerrit-Change-Number: 14349 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 16:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8998: admission control accounting for mt dop
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14357 to look at the new patch set (#5). Change subject: WIP: IMPALA-8998: admission control accounting for mt_dop .. WIP: IMPALA-8998: admission control accounting for mt_dop This integrates mt_dop with the "slots" mechanism that's used for non-default executor groups. The idea is simple - the degree of parallelism on a backend determines the number of slots consumed. The effective degree of parallelism is used, not the raw mt_dop setting. E.g. if the query only has a single input split and executes only a single fragment instance on a host, we don't want to count the full mt_dop value for admission control. The admission control logic is extended to take this into account. We also add an immediate rejection code path since it is now possible for queries to not be admittable based on the # of available slots. We only factor in the "width" of the plan - i.e. the number of instances of fragments. We don't account for the number of distinct fragments, since they may not actually execute in parallel with each other because of dependencies. This number is added to the per-host profile as the "AdmissionSlots" counter. Testing: Added unit tests for rejection and queue/admit checks. Added end-to-end tests: * test_admission_slots in test_mt_dop.py that checks the admission slot calculation via the profile. * TODO: End-to-end admission test Added checks to test_verify_metrics (which runs after end-to-end tests) to ensure that the per-backend slots in use goes to 0 when the cluster is quiesced. Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/StatestoreService.thrift A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_mt_dop.py M tests/verifiers/test_verify_metrics.py M www/backends.tmpl 16 files changed, 311 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/5 -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] WIP: IMPALA-8998: admission control accounting for mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14357 ) Change subject: WIP: IMPALA-8998: admission control accounting for mt_dop .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/14357/5/tests/query_test/test_mt_dop.py File tests/query_test/test_mt_dop.py: http://gerrit.cloudera.org:8080/#/c/14357/5/tests/query_test/test_mt_dop.py@123 PS5, Line 123: @SkipIfNotHdfsMinicluster.tuned_for_minicluster flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/14357/5/tests/verifiers/test_verify_metrics.py File tests/verifiers/test_verify_metrics.py: http://gerrit.cloudera.org:8080/#/c/14357/5/tests/verifiers/test_verify_metrics.py@61 PS5, Line 61: a flake8: F841 local variable 'all_zero' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/14357/5/tests/verifiers/test_verify_metrics.py@62 PS5, Line 62: t flake8: F841 local variable 'timeout_expired' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/14357/5/tests/verifiers/test_verify_metrics.py@65 PS5, Line 65: flake8: E261 at least two spaces before inline comment -- To view, visit http://gerrit.cloudera.org:8080/14357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5 Gerrit-Change-Number: 14357 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 03 Oct 2019 16:33:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5046/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 16:32:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8742: Switch to ScanRange::bytes to read() instead of len()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14348 ) Change subject: IMPALA-8742: Switch to ScanRange::bytes_to_read() instead of len() .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5045/ -- To view, visit http://gerrit.cloudera.org:8080/14348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie896db3f4b5f3e2272d81c2d360049af09c41d9c Gerrit-Change-Number: 14348 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 16:29:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8498: Write column index for floating types when NaN is not present
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14264 ) Change subject: IMPALA-8498: Write column index for floating types when NaN is not present .. Patch Set 6: Code-Review+1 (4 comments) http://gerrit.cloudera.org:8080/#/c/14264/6/be/src/exec/parquet/hdfs-parquet-table-writer.cc File be/src/exec/parquet/hdfs-parquet-table-writer.cc: http://gerrit.cloudera.org:8080/#/c/14264/6/be/src/exec/parquet/hdfs-parquet-table-writer.cc@436 PS6, Line 436: //UNLIKELY( remove? http://gerrit.cloudera.org:8080/#/c/14264/6/be/src/exec/parquet/hdfs-parquet-table-writer.cc@470 PS6, Line 470: // IMPALA-7304: Don't write column index for floating-point columns until : // PARQUET-1222 is resolved. This is modified by: : // IMPALA-8498: Write column index for floating types when NaN is not present nit: I think that mentioning IMPALA-7304 just make this comment more confusing. http://gerrit.cloudera.org:8080/#/c/14264/6/tests/query_test/test_parquet_page_index.py File tests/query_test/test_parquet_page_index.py: http://gerrit.cloudera.org:8080/#/c/14264/6/tests/query_test/test_parquet_page_index.py@305 PS6, Line 305: nit: extra line (we generally leave only 1 line between functions) http://gerrit.cloudera.org:8080/#/c/14264/6/tests/query_test/test_parquet_page_index.py@466 PS6, Line 466: unique_database, no_nan_tbl, : "(1.5), (2.3), (4.5), (42.42), (3.1415), (0.0)") here and at several other lines: needs + 4 indentation -- To view, visit http://gerrit.cloudera.org:8080/14264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9d367500243c8ca142a16ebfeef6c841f013434 Gerrit-Change-Number: 14264 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 03 Oct 2019 15:56:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/14291 ) Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2 .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/14291/3/tests/query_test/test_cast_with_format.py File tests/query_test/test_cast_with_format.py: http://gerrit.cloudera.org:8080/#/c/14291/3/tests/query_test/test_cast_with_format.py@735 PS3, Line 735: '1985-\some text12-08 Please use a raw string here too, to make it explicit if the backslash means a literal backslash or an escape character. -- To view, visit http://gerrit.cloudera.org:8080/14291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855 Gerrit-Change-Number: 14291 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 15:32:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/14291 )
Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14291/3/tests/query_test/test_cast_with_format.py
File tests/query_test/test_cast_with_format.py:
http://gerrit.cloudera.org:8080/#/c/14291/3/tests/query_test/test_cast_with_format.py@724
PS3, Line 724: format '-"some \ text"MM-DD')
I'm confused about the usage of backslash in this format string and the format
string in L727.
Is it used for a literal backslash character or for escaping the space
character that follows it?
1. For instance a slightly modified select:
select cast("1985- some text12-05" as date format '-"some \ text"MM-DD');
Returns : 1985-12-05
Which makes me think that the backslash in the format string escapes the space.
2. On the other hand the select in L726
select cast(cast("1985-12-06" as date) as string format '-"some \
text"MM-DD')
Returns: 1985-some \ text12-06
Which makes me think that the backslash in the format string is supposed to be
taken literally.
http://gerrit.cloudera.org:8080/#/c/14291/3/tests/query_test/test_cast_with_format.py@730
PS3, Line 730: '
Probably the apostrophe can be removed for clarity. I assume the - separator in
the format string matches -' in the input string.
http://gerrit.cloudera.org:8080/#/c/14291/3/tests/query_test/test_cast_with_format.py@731
PS3, Line 731: -"\some text"MM-DD')
Again, the backslash is used here to escape the s character right?
However the backslash in the format string in L714 is interpreted literally.
It could be interesting to try running these with Hive and check how it handles
backslashes.
--
To view, visit http://gerrit.cloudera.org:8080/14291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855
Gerrit-Change-Number: 14291
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 03 Oct 2019 15:15:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8498: Write column index for floating types when NaN is not present
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14264 ) Change subject: IMPALA-8498: Write column index for floating types when NaN is not present .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4715/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9d367500243c8ca142a16ebfeef6c841f013434 Gerrit-Change-Number: 14264 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 03 Oct 2019 14:25:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Change supported year range for DATE values to 1..9999
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14349 ) Change subject: IMPALA-7368: Change supported year range for DATE values to 1.. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iefdf1c036834763f52d44d0c39a25a1f04e41e07 Gerrit-Change-Number: 14349 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 13:56:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8996: fix test show create table in test zorder.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14359 ) Change subject: IMPALA-8996: fix test_show_create_table in test_zorder.py .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4714/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic316224325eec64d9b1e570854a74c0372084a4a Gerrit-Change-Number: 14359 Gerrit-PatchSet: 2 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 03 Oct 2019 13:52:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8498: Write column index for floating types when NaN is not present
Norbert Luksa has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/14264 ) Change subject: IMPALA-8498: Write column index for floating types when NaN is not present .. IMPALA-8498: Write column index for floating types when NaN is not present IMPALA-7307 disabled column index writing for floating point columns until PARQUET-1222 is resolved. However, the problematic values are only the NaNs. Therefore we can write column index if NaNs are not present in data. Testing: * Added tests which should fail if a column index is present while having NaN values in the column. Change-Id: Ic9d367500243c8ca142a16ebfeef6c841f013434 --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M tests/query_test/test_parquet_page_index.py 2 files changed, 110 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/14264/6 -- To view, visit http://gerrit.cloudera.org:8080/14264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic9d367500243c8ca142a16ebfeef6c841f013434 Gerrit-Change-Number: 14264 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
