[Impala-ASF-CR] IMPALA-8642: krpc should depend on thrift-deps in CMake
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13569 ) Change subject: IMPALA-8642: krpc should depend on thrift-deps in CMake .. IMPALA-8642: krpc should depend on thrift-deps in CMake When build with IMPALA_BUILD_THREADS=1, the compiling of krpc will fail since kudu_version.cc includes "gen-cpp/Status_types.h" which will be built later. We should add dependency on thrift-deps for krpc to avoid such a race condition in compiling. Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Reviewed-on: http://gerrit.cloudera.org:8080/13569 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/kudu/rpc/CMakeLists.txt 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Gerrit-Change-Number: 13569 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3558/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 11 Jun 2019 01:43:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 8: Code-Review+2 Rebased. Carry Todd's +2. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 11 Jun 2019 01:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4441/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 11 Jun 2019 01:02:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/8 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Bump CDP_BUILD_NUMBER to 1153860 This patch bumps the CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Reviewed-on: http://gerrit.cloudera.org:8080/13574 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 5 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 11 Jun 2019 00:49:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1)
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13522 ) Change subject: IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1) .. Patch Set 4: Code-Review+2 (3 comments) Thanks for fixing it. http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/runtime/io/handle-cache.inline.h File be/src/runtime/io/handle-cache.inline.h: http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/runtime/io/handle-cache.inline.h@93 PS4, Line 93: DCHECK_GT(mtime, 0); super minor nit: is 0 invalid ? http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/runtime/io/request-ranges.h File be/src/runtime/io/request-ranges.h: http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/runtime/io/request-ranges.h@170 PS4, Line 170: 'mtime' matches : // the modified time of the cached file in the HDFS cache stale http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/runtime/io/request-ranges.h@244 PS4, Line 244: int64_t mtime, Nice to have one comment about it. -- To view, visit http://gerrit.cloudera.org:8080/13522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672 Gerrit-Change-Number: 13522 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 23:31:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8642: krpc should depend on thrift-deps in CMake
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13569 ) Change subject: IMPALA-8642: krpc should depend on thrift-deps in CMake .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Gerrit-Change-Number: 13569 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 23:20:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8642: krpc should depend on thrift-deps in CMake
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13569 ) Change subject: IMPALA-8642: krpc should depend on thrift-deps in CMake .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4440/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Gerrit-Change-Number: 13569 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 23:20:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8642: krpc should depend on thrift-deps in CMake
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13569 ) Change subject: IMPALA-8642: krpc should depend on thrift-deps in CMake .. Patch Set 1: OK. Let's do it in the future :) -- To view, visit http://gerrit.cloudera.org:8080/13569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Gerrit-Change-Number: 13569 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 23:20:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 23:10:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. IMPALA-8623: Expose HS2 HTTP port in containers Testing: Ran dockerised test cluster locally, checked that ports were mapped as expected. Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Reviewed-on: http://gerrit.cloudera.org:8080/13520 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M docker/coord_exec/Dockerfile M docker/coordinator/Dockerfile 2 files changed, 4 insertions(+), 0 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/13307 )
Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission
Controller.
..
Patch Set 8:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/13307/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13307/3//COMMIT_MSG@15
PS3, Line 15: Max Running
> OK, this is complicated, like all naming things.
The reason I was vouching for requests because then it will be consistent with
the other config param (max_requests) which it is directly related to, both of
these params will be written to the same file, so having them consistent
locally will be less confusing.
I do however completely agree that "queries" will be more clearer so lets stick
to that.
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc
File be/src/scheduling/admission-controller-test.cc:
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@317
PS8, Line 317: CheckRoundingForPool(ac, /*expected*/ 0, /*parameter*/ 0, /*num
hosts*/ 100)
nit: repeated test
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@323
PS8, Line 323: CheckRoundingForPool(ac, /*expected*/ 1, /*parameter*/
100, /*num hosts*/ 100);
how about another one: CheckRoundingForPool(ac, /*expected*/ 2, /*parameter*/
.5, /*num hosts*/ 3);
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@327
PS8, Line 327: CanAdmitRequest
this test is only testing the functionality inside HasAvailableMemResources(),
can you add another test which manipulates the pool_stats to test out the other
num_running queries scalable param.
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@353
PS8, Line 353: // Check that the query can be admitted.
: string not_admitted_reason;
: ASSERT_TRUE(admission_controller->CanAdmitRequest(
: *query_schedule, config_d, host_count, true,
_admitted_reason));
:
: // The query scales with cluster size of 1000.
: host_count = 1000;
: ASSERT_TRUE(admission_controller->CanAdmitRequest(
: *query_schedule, config_d, host_count, true,
_admitted_reason));
nit: would it make sense if in the first case host_count=2 the query cannot be
updated and in the second case it can because it scaled with the increase in
cluster size
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.h
File be/src/scheduling/admission-controller.h:
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.h@294
PS8, Line 294: /// Pointer to the cluster membership manager.
nit: mention that it is not owned by this class
http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1403
PS3, Line 1403: string, pair I don;t understand what you mean here. Do you mean QuerySchedule.per_backen
pool_config.max_memory_multiple represents the max memory per node allocated to
this resource pool. Now if that value is more than an executor's
TBackendDescriptor.admit_mem_limit then we are effectively over-committing
memory for this pool on that node
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc@484
PS8, Line 484: *not_admitted_reason = Substitute(QUEUED_QUEUE_NOT_EMPTY,
: stats->local_stats().num_queued,
GetStalenessDetailLocked(" "));
: return false;
: } else if (max_requests >= 0 && stats->agg_num_running() >=
max_requests) {
: *not_admitted_reason = Substitute(QUEUED_NUM_RUNNING,
stats->agg_num_running(),
: max_requests, GetStalenessDetailLocked(" "));
nit: should we add info similar to GetMaxQueuedForPoolDescription here for both
QUEUED_QUEUE_NOT_EMPTY and QUEUED_NUM_RUNNING?
http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc@1422
PS8, Line 1422: If this
: // value could change between the time a query is queued and
the time that we try
: // to admit it from the queue then it might be impossible to
run the query at the
: // head of the queue, which might block admission forever.
nit: how about=> If we exclude the quiescing executors and a query at the head
of the queue is scheduled to run on those, then the scaled down limits of the
pool would prevent it from being admitted and hold up the rest of the queue
[Impala-ASF-CR] Add option to set driver in ImpalaJdbcDriver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13514 ) Change subject: Add option to set driver in ImpalaJdbcDriver .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Gerrit-Change-Number: 13514 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 21:56:52 + Gerrit-HasComments: No
[Impala-ASF-CR] Add option to set driver in ImpalaJdbcDriver
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13514 ) Change subject: Add option to set driver in ImpalaJdbcDriver .. Add option to set driver in ImpalaJdbcDriver ImpalaJdbcDriver is a simple class that provides a wrapper around the Hive Jdbc driver for use in testing. This patch adds a '-d' parameter that takes a driver class and executes queries with that driver instead of the Hive driver, if its in the classpath. This makes it easy to use ./bin/run-jdbc-client.sh to test other Jdbc drivers, such as the proprietary Cloudera Impala driver. Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Reviewed-on: http://gerrit.cloudera.org:8080/13514 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins --- M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java 1 file changed, 26 insertions(+), 14 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Gerrit-Change-Number: 13514 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 8: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/13307/6/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13307/6/be/src/scheduling/admission-controller.cc@1445 PS6, Line 1445: > Yes! I see you added a regression test - thank you! -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 21:37:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1)
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13522 ) Change subject: IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1) .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/13522/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13522/4//COMMIT_MSG@28 PS4, Line 28: - Passed core tests > Did you do any testing on S3 or other object stores? Not sure what the mtim I'm running S3 tests. Nothing failing so far, so I think the DCHECKs are ok on S3. I think the big question is whether the DCHECK in ScanRange::Reset() is overly restrictive for anything like ADLS or ABFS. http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/exec/hdfs-scan-node-base.h File be/src/exec/hdfs-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/13522/4/be/src/exec/hdfs-scan-node-base.h@278 PS4, Line 278: /// Old API for compatibility with text scanners (e.g. LZO text scanner). > Maybe we can remove this one now? Since I think the version of the LZO scan hdfs-lzo-text-scanner.cc still uses this one. It doesn't use BufferOpts right now. BufferOpts is now mainly for try_cache, so it seems like LZO could use one of the new ones if it used BufferOpts. -- To view, visit http://gerrit.cloudera.org:8080/13522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672 Gerrit-Change-Number: 13522 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 21:15:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1)
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13522 ) Change subject: IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1) .. Patch Set 4: Code-Review+2 I think this is ok, I'm not sure that either of my comments is a blocker since this is a strict improvement anyway, but it'd be good to know the answer. -- To view, visit http://gerrit.cloudera.org:8080/13522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672 Gerrit-Change-Number: 13522 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 21:01:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Sudhanshu Arora has posted comments on this change. (
http://gerrit.cloudera.org:8080/13558 )
Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@78
PS1, Line 78: private static final String CONNECTORREAD =
"CONNECTORREAD".intern();
How about using enum for this?
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java@93
PS1, Line 93: analyzer.ensureTableWriteSupported(table_);
Nit: Another way to do this is to introduce another enum say OperationType.
public enum OperationType {
READ,
WRITE,
METADATA_MODIFICATION
}
And then mark Alter Table as METADATA_MODIFICATION.
Then we can have a function in Analyzer that takes the operationType and tells
whether the operation is allowed on the table or not.
That will make this cleaner I think
Or we can introduce annotations like
@SupportedTableTypes
@OperationType
and we can say that every stmt node must be annotated with these. That will
make it clear.
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@129
PS1, Line 129: public static final byte REQUIRE_READ = (byte)2;
If we create an enum then these values can go into enum itself
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
File fe/src/test/java/org/apache/impala/analysis/ParserTest.java:
http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@4021
PS1, Line 4021: ParserError("Create table bucketed_tbl(order_id int,
order_name string)"
Wouldn't this fail during analysis?
--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 10 Jun 2019 20:54:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 4: (1 comment) Thanks for creating a test case for non-existent entities. Patch looks good to me other than a minor comment below. http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@755 PS4, Line 755: createFakeInsertEvent I see you are creating the insert events by calling the HMS API. You can probably remove this method? -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 10 Jun 2019 20:46:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Update mustache to commit b290952d8eb93d085214d8c8c9eab8559df9f606
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13563 ) Change subject: Update mustache to commit b290952d8eb93d085214d8c8c9eab8559df9f606 .. Patch Set 1: Code-Review+2 I checked this out and clicked around the debug pages a bit to make sure everything looked normal -- To view, visit http://gerrit.cloudera.org:8080/13563 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If26f3aaa2a3279a1f6a300c4f4cee7ec899e22ed Gerrit-Change-Number: 13563 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 20:15:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3557/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 20:04:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py@572 PS6, Line 572: assert "Error granting a privilege in Ranger. Ranger error message: " \ > don't these asserts need to be updated? Sorry, I forgot to commit this change. Done. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:25:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/7 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:22:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4439/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:22:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13548 )
Change subject: IMPALA-8633 : Insert event should not error when table does not
exists
..
Patch Set 4:
(1 comment)
Is there any e2e test we could write to stress these paths? eg if you were to
run the metastore poller with a 5 second frequency, and you did some random
sequence of hive operations, would it end up triggering it? I'm wondering
whether we have similar issues in other events. For example, if you ALTER TABLE
followed by DROP, will Impala get desynchronized?
http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@829
PS4, Line 829: }
should this have an else { throw e }?
--
To view, visit http://gerrit.cloudera.org:8080/13548
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693
Gerrit-Change-Number: 13548
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Mon, 10 Jun 2019 19:10:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] fe: improve logging for metadata loading
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13463 ) Change subject: fe: improve logging for metadata loading .. Patch Set 1: Vihang, any chance you can take a look at this? -- To view, visit http://gerrit.cloudera.org:8080/13463 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d90a5f075b05d2dc96692b3349abe35ce24b8b8 Gerrit-Change-Number: 13463 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 10 Jun 2019 19:11:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py@572 PS6, Line 572: assert "Error granting a privilege to Ranger. Ranger error message: " \ don't these asserts need to be updated? -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:03:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:01:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Include partition id when calculating consistent remote placement
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Include partition id when calculating consistent remote placement .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13545/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13545/1//COMMIT_MSG@17 PS1, Line 17: This adds the partition_id in the hash, so files with the same > How are the partition IDs generated? I'm wondering if there might be a less They are just sequential numbers that we make up, but it turns out that they are unique, even across tables: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java#L572 But that creates a different problem, which is that they are not stable over time or across different coordinators. If I invalidate metadata and rerun a query, I get different partition ids. I'll explore ways to make something stable to use. -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 18:31:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 10: (6 comments) Mainly comments about comments and some cleanup. One administrative thing - it would have been a little easier to review PS10 if you did the rebase in a separate patchset. The diff from PS9->PS10 was noisy because of unrelated changes in query-options.cc picked up from the rebase. http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@179 PS10, Line 179: private static double ESTIMATED_COMPRESSION_FACTOR_LEGACY = 3.58;// to change Can you remove these "to change" comments? I don't think they help much. http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java: http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@225 PS10, Line 225: // functional.alltypesmixedformat is a table of 4 partitions, I have some nits about these test comments. I think they should be javadoc comments, just for consistency. The text is also wrapped at < 90 lines - in some cases the comment would fit on fewer lines. http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@243 PS10, Line 243: // True cardinality of tpch_text_gzip.lineitem is 6,001,215. Thanks for these comments about the cardinality, this is actually really helpful to understand the test. http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@516 PS10, Line 516: // Estimated cardinality of the NestedLoopJoinNode is 550,564 = 742 * 742. I guess we're not so good at estimating cardinality of tiny parquet files because of the footer? http://gerrit.cloudera.org:8080/#/c/12974/10/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@564 PS10, Line 564: //TODO: It seems that the cardinality of the SelectNode should be 1 instead Nice catch. Can you file a JIRA for this and mention in TODO? This is much better for tracking purposes - bugs that are only tracked by TODOs in the code tend to be forgotten easily. And consider fixing it in a separate commit. It seems like an oversight, I think this code should probably have a max(1, ...) to avoid setting it to 0. cardinality_ = Math.round(((double) getChild(0).cardinality_) * computeSelectivity()); Preconditions.checkState(cardinality_ >= 0); http://gerrit.cloudera.org:8080/#/c/12974/10/testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-disabled.test File testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-disabled.test: PS10: I think you can revert this file to its original name (same for the other similar cases where gerrit shows a rename). Encoding all of the options in the file name isn't really scalable, so I'd prefer default-join-distr-mode-shuffle.test for conciseness. -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 18:07:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3556/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 18:04:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3555/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 17:51:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4438/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 17:25:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. Patch Set 3: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 17:25:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Hello Thomas Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13520 to look at the new patch set (#3). Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. IMPALA-8623: Expose HS2 HTTP port in containers Testing: Ran dockerised test cluster locally, checked that ports were mapped as expected. Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 --- M docker/coord_exec/Dockerfile M docker/coordinator/Dockerfile 2 files changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/13520/3 -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13520/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13520/2//COMMIT_MSG@10 PS2, Line 10: JDBC HTTP tests are not currently working. Once IMPALA-8626 is fixed, : they should exercise this. outdated -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 17:21:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Andrew Sherman has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. Add 3 configuration parameters to Admission Controller that scale with the number of hosts in the resource pool. These parameters are specified to the Impalad through the -llama_site_path flag which points to a Llama XML configuration file. The new configuration parameters are: + Max Running Queries Multiple - this floating point number is multiplied by the current total number of executors at runtime to give the maximum number of concurrently running queries allowed in the pool. This calculation is rounded up to the nearest integer so the result will always be at least one as long as the parameter is non-zero. + Max Queued Queries Multiple - this floating point number is multiplied by the current total number of executors at runtime to give the maximum number of queries that can be queued in the pool. This calculation is rounded up to the nearest integer so the result will always be at least one as long as the parameter is non-zero. + Max Memory Multiple - this number of bytes is multiplied by the current total number of executors at runtime to give the maximum memory available across the cluster for the pool. If any of these parameters have zero value then they will be ignored. In this case the corresponding non-scalable parameters will be used, if they are set. The new parameters are exposed through the webui. At various points in the code Admission Controller looks at the Pool Config objects to find non-scalable parameters such as the max number of queries that can run in the pool. These access have been encapsulated in functions that return the scalable version of the configuration value if the new scalable parameters are being used. Diagnostic messages are enhanced to show the origin of the encapsulated parameters. TESTING All end-to-end tests are running clean with ASAN. The unit test admission-controller-test.cc has been expanded to test the newly added code. Added an end-to-end test that adds and removes Impalads from a minicluster. Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_metrics.py M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/util/RequestPoolService.java M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java M fe/src/test/resources/fair-scheduler-test.xml M fe/src/test/resources/fair-scheduler-test2.xml M fe/src/test/resources/llama-site-test.xml M fe/src/test/resources/llama-site-test2.xml M testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M tests/common/impala_cluster.py M tests/custom_cluster/test_admission_controller.py M tests/webserver/test_web_pages.py M www/admission_controller.tmpl 20 files changed, 1,338 insertions(+), 297 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13307/8 -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3553/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 16:45:54 + Gerrit-HasComments: No
[Impala-ASF-CR] Add option to set driver in ImpalaJdbcDriver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13514 ) Change subject: Add option to set driver in ImpalaJdbcDriver .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4437/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Gerrit-Change-Number: 13514 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 16:24:23 + Gerrit-HasComments: No
[Impala-ASF-CR] Add option to set driver in ImpalaJdbcDriver
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13514 ) Change subject: Add option to set driver in ImpalaJdbcDriver .. Patch Set 4: Code-Review+2 gvo failure was unrelated -- To view, visit http://gerrit.cloudera.org:8080/13514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Gerrit-Change-Number: 13514 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 16:24:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/6 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13525 )
Change subject: IMPALA-8551: Make the grant/revoke error messages to be more
user friendly
..
Patch Set 6:
(2 comments)
> Patch Set 5:
>
> (2 comments)
>
> Looks pretty good modulo some nits. One thing, though: would it be possible
> to split this into one commit to bump the CDP_BUILD_NUMBER and one to change
> the error messages? I'm afraid when we look through lists of commits to
> backport across branches, we'll see this and think "oh it's just a CDP
> version bump, we can skip this one and pick up a later one" whereas in fact
> it also contains a functional change.
Split into two now.
https://gerrit.cloudera.org/c/13574/
https://gerrit.cloudera.org/c/13525/
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@173
PS5, Line 173: try (AutoFlush auditHandler =
RangerBufferAuditHandler.autoFlush()) {
> pre-existing issue, but: is it possible to add a DEBUG level log message he
Wouldn't be better to use ERROR log level instead?
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@174
PS5, Line 174: plugin_.get().grantAccess(request, auditHandler);
> nit: can we reword this to "in Ranger" instead of "to Ranger" -- the privil
Done
--
To view, visit http://gerrit.cloudera.org:8080/13525
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576
Gerrit-Change-Number: 13525
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 10 Jun 2019 16:17:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8623: Expose HS2 HTTP port in containers
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13520 ) Change subject: IMPALA-8623: Expose HS2 HTTP port in containers .. Patch Set 2: Did you still want to get this in, given the conclusion we reached on IMPALA-8626? -- To view, visit http://gerrit.cloudera.org:8080/13520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iece20bc134fa5867f18b166cee2a2f75b21f9f36 Gerrit-Change-Number: 13520 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 10 Jun 2019 16:13:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8629: (part 1) Add temp KuduStorageHandler
Thomas Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13561 ) Change subject: IMPALA-8629: (part 1) Add temp KuduStorageHandler .. IMPALA-8629: (part 1) Add temp KuduStorageHandler This patch adds a temporary KuduStorageHandler so that the Kudu project can change its handler without breaking the integration. It also disabled any tests that depend on the specific handler. A follow up patch will remove the TEMP_KUDU_STORAGE_HANDLER. adjust the KUDU_STORAGE_HANDLER value to be the final value, and re-enable the tests. Change-Id: Ic9982466699818390fa28efc5ea1aae75b11c12a Reviewed-on: http://gerrit.cloudera.org:8080/13561 Tested-by: Impala Public Jenkins Reviewed-by: Thomas Marshall --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 4 files changed, 24 insertions(+), 13 deletions(-) Approvals: Impala Public Jenkins: Verified Thomas Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic9982466699818390fa28efc5ea1aae75b11c12a Gerrit-Change-Number: 13561 Gerrit-PatchSet: 4 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8629: (part 1) Add temp KuduStorageHandler
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13561 ) Change subject: IMPALA-8629: (part 1) Add temp KuduStorageHandler .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9982466699818390fa28efc5ea1aae75b11c12a Gerrit-Change-Number: 13561 Gerrit-PatchSet: 3 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 10 Jun 2019 16:12:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 1: Splitting the https://gerrit.cloudera.org/c/13525/ into two commits. -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 16:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13574 Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Bump CDP_BUILD_NUMBER to 1153860 This patch bumps the CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b --- M bin/impala-config.sh 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13574/1 -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13525 )
Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860
..
Patch Set 5:
(2 comments)
Looks pretty good modulo some nits. One thing, though: would it be possible to
split this into one commit to bump the CDP_BUILD_NUMBER and one to change the
error messages? I'm afraid when we look through lists of commits to backport
across branches, we'll see this and think "oh it's just a CDP version bump, we
can skip this one and pick up a later one" whereas in fact it also contains a
functional change.
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@173
PS5, Line 173: } catch (Exception e) {
pre-existing issue, but: is it possible to add a DEBUG level log message here
which includes the stack trace of the exception? I'm afraid at some point we'll
trigger an NPE or something within the Ranger client code, and the excepption
will get swallowed, and all we'll see is "null" in the error message.
(same below)
http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@174
PS5, Line 174: throw new InternalException("Error granting a privilege to
Ranger. " +
nit: can we reword this to "in Ranger" instead of "to Ranger" -- the privilege
is being granted to some target user, not to the ranger service itself.
--
To view, visit http://gerrit.cloudera.org:8080/13525
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576
Gerrit-Change-Number: 13525
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 10 Jun 2019 15:50:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8642: krpc should depend on thrift-deps in CMake
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13569 ) Change subject: IMPALA-8642: krpc should depend on thrift-deps in CMake .. Patch Set 1: We don't have that (it would be painfully slow to do from scratch). You could do a once-off test by hacking impala-config.sh or similar. The natural place to put it would be at the end of bin/jenkins/build-all-flag-combinations.sh, since at least then ccache would be warmed up. It would still be pretty slow and I'm not sure it's worth slowing down precommit. My inclination would be to not add it at this point, and only add it later if it turns out to be a recurring issue. -- To view, visit http://gerrit.cloudera.org:8080/13569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Gerrit-Change-Number: 13569 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 15:06:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. Patch Set 1: Please help me review the patch. Thanks -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 10 Jun 2019 10:11:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8642: krpc should depend on thrift-deps in CMake
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13569 ) Change subject: IMPALA-8642: krpc should depend on thrift-deps in CMake .. Patch Set 1: Hi Tim, Is there a Jenkins job that I can test for building with one thread? Or can we add a parameter for IMPALA_BUILD_THREADS in the "ubuntu-16.04-build-only" job? We'd better add this check into the "gerrit-code-review-checks" job too, so that we can avoid introducing such kind of bugs in the future. -- To view, visit http://gerrit.cloudera.org:8080/13569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaf9d992340f07fdf88b35fb967553fa2e35b628 Gerrit-Change-Number: 13569 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 09:31:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: signed overflow is undefined behavior
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13437 ) Change subject: IMPALA-5031: signed overflow is undefined behavior .. IMPALA-5031: signed overflow is undefined behavior Fix remaining signed overflow undefined behaviors in end-to-end tests. The interesting part of the backtraces: exprs/aggregate-functions-ir.cc:464:25: runtime error: signed integer overflow: 0x5a4728ca063b522c0b728f80 + 0x3c2f7086aed236c807a1b500 cannot be represented in type '__int128' #0 AggregateFunctions::DecimalAvgMerge( impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal*) exprs/aggregate-functions-ir.cc:464:25 #1 AggFnEvaluator::Update(TupleRow const*, Tuple*, void*) exprs/agg-fn-evaluator.cc:327:7 #2 AggFnEvaluator::Add(TupleRow const*, Tuple*) exprs/agg-fn-evaluator.h:257:3 #3 Aggregator::UpdateTuple(AggFnEvaluator**, Tuple*, TupleRow*, bool) exec/aggregator.cc:167:24 #4 NonGroupingAggregator::AddBatchImpl(RowBatch*) exec/non-grouping-aggregator-ir.cc:27:5 #5 NonGroupingAggregator::AddBatch(RuntimeState*, RowBatch*) exec/non-grouping-aggregator.cc:124:45 #6 AggregationNode::Open(RuntimeState*) exec/aggregation-node.cc:70:57 exprs/aggregate-functions-ir.cc:513:12: runtime error: signed integer overflow: -8282081183197145958 + -4473782455107795527 cannot be represented in type 'long' #0 void AggregateFunctions::SumUpdate(impala_udf::FunctionContext*, impala_udf::BigIntVal const&, impala_udf::BigIntVal*) exprs/aggregate-functions-ir.cc:513:12 #1 AggFnEvaluator::Update(TupleRow const*, Tuple*, void*) exprs/agg-fn-evaluator.cc:327:7 #2 AggFnEvaluator::Add(TupleRow const*, Tuple*) exprs/agg-fn-evaluator.h:257:3 #3 Aggregator::UpdateTuple(AggFnEvaluator**, Tuple*, TupleRow*, bool) exec/aggregator.cc:167:24 #4 NonGroupingAggregator::AddBatchImpl(RowBatch*) exec/non-grouping-aggregator-ir.cc:27:5 #5 NonGroupingAggregator::AddBatch(RuntimeState*, RowBatch*) exec/non-grouping-aggregator.cc:124:45 #6 AggregationNode::Open(RuntimeState*) exec/aggregation-node.cc:70:57 exprs/aggregate-functions-ir.cc:585:14: runtime error: signed integer overflow: 0x5a4728ca063b522c0b728f80 + 0x3c2f7086aed236c807a1b500 cannot be represented in type '__int128' #0 AggregateFunctions::SumDecimalMerge( impala_udf::FunctionContext*, impala_udf::DecimalVal const&, impala_udf::DecimalVal*) exprs/aggregate-functions-ir.cc:585:14 #1 AggFnEvaluator::Update(TupleRow const*, Tuple*, void*) exprs/agg-fn-evaluator.cc:327:7 #2 AggFnEvaluator::Add(TupleRow const*, Tuple*) exprs/agg-fn-evaluator.h:257:3 #3 Aggregator::UpdateTuple(AggFnEvaluator**, Tuple*, TupleRow*, bool) exec/aggregator.cc:167:24 #4 NonGroupingAggregator::AddBatchImpl(RowBatch*) exec/non-grouping-aggregator-ir.cc:27:5 #5 NonGroupingAggregator::AddBatch(RuntimeState*, RowBatch*) exec/non-grouping-aggregator.cc:124:45 #6 AggregationNode::Open(RuntimeState*) exec/aggregation-node.cc:70:57 runtime/decimal-value.inline.h:145:12: runtime error: signed integer overflow: 18 * 0x0785ee10d5da46d900f436a0 cannot be represented in type '__int128' #0 DecimalValue<__int128>::ScaleTo(int, int, int, bool*) const runtime/decimal-value.inline.h:145:12 #1 DecimalOperators::ScaleDecimalValue( impala_udf::FunctionContext*, DecimalValue const&, int, int, int) exprs/decimal-operators-ir.cc:132:41 #2 DecimalOperators::RoundDecimal(impala_udf::FunctionContext*, impala_udf::DecimalVal const&, int, int, int, int, DecimalOperators::DecimalRoundOp const&) exprs/decimal-operators-ir.cc:465:16 #3 DecimalOperators::RoundDecimal(impala_udf::FunctionContext*, impala_udf::DecimalVal const&, DecimalOperators::DecimalRoundOp const&) exprs/decimal-operators-ir.cc:519:10 #4 DecimalOperators::CastToDecimalVal( impala_udf::FunctionContext*, impala_udf::DecimalVal const&) exprs/decimal-operators-ir.cc:529:10 #5 impala_udf::DecimalVal ScalarFnCall::InterpretEval (ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:208 #6 ScalarFnCall::GetDecimalVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:618:44 #7 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:321:27 #8 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:251:10 #9 Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow service/fe-support.cc:246:26 #10 ()
[Impala-ASF-CR] IMPALA-5031: signed overflow is undefined behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13437 ) Change subject: IMPALA-5031: signed overflow is undefined behavior .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73dd6802ec1023275d09a99a2950f3558313fc8e Gerrit-Change-Number: 13437 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 10 Jun 2019 08:34:26 + Gerrit-HasComments: No
