[Impala-ASF-CR] IMPALA-8665 Include extra info in error message when date cast fails
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/13680 )
Change subject: IMPALA-8665 Include extra info in error message when date cast
fails
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc
File be/src/exprs/cast-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc@310
PS2, Line 310: DateVal CastFunctions::CastToDateVal(FunctionContext* ctx, const
StringVal& val) {
> I have already tested all the tests you mentioned. They all passed. The way
Yes, please extend the affected tests to cover the whole error message.
Update: Nevermind, I see you enhanced the tests. Thanks!
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc@316
PS2, Line 316: >SetError(
> It's not working because StringVal.ptr is not a null-terminated string.
Thanks for the explanation.
http://gerrit.cloudera.org:8080/#/c/13680/3/testdata/workloads/functional-query/queries/QueryTest/date.test
File testdata/workloads/functional-query/queries/QueryTest/date.test:
http://gerrit.cloudera.org:8080/#/c/13680/3/testdata/workloads/functional-query/queries/QueryTest/date.test@125
PS3, Line 125: not a date
I wonder if it would be more readable to put the incorrect input within
quotation marks here.
For example it would help with this input:
select CAST(" " AS DATE);
UDF ERROR: String to Date parse failed. Invalid string val:
--
To view, visit http://gerrit.cloudera.org:8080/13680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04
Gerrit-Change-Number: 13680
Gerrit-PatchSet: 3
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Comment-Date: Fri, 21 Jun 2019 06:59:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3708/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 21 Jun 2019 06:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories The FileMetadataLoader is used to load the file information in when the table is loaded. By default, it lists all the files in the table/partition directory. Currently, it only skips the filenames which are invalid (hidden files and ones starting with "_" etc). However, it does not skip the directories which are temporary or hidden. In case of Hive when data is inserted into a table, it creates a temporary staging directory which is a hidden directory under the table location. When the insert in hive is completed, such staging directories are removed. But if there is a refresh called during that time, FileMetadataLoader will add the files in the staging directory as well. Not only this could cause temporary invalid results but it causes table to go in a bad state when these temporary directories are removed. The only work-around in such a case to issue a refresh on the table again. This patch adds logic in the filemetadataloader to ignore such temporary staging directories. Unfortunately, hadoop does not provide a API which can recursively list files in a directory and skip certain directories. This patch addes this logic of filtering into existing RecursingIterator in FileSystemUtil. Testing: 1. Added a new test in FilemetadataloaderTest and modified existing ones in AcidUtils 2. Ran concurrent inserts from Hive while issuing refresh in a loop on Impala side. Earlier this would cause the table to go into a bad state. Now, it works fine for the staging directories. It still runs into a FileNotFoundException from the impalad when there are insert overwrite statements in Hive Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java 3 files changed, 96 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13665/5 -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13665 )
Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp
directories
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13665/4/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/13665/4/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@566
PS4, Line 566: if (isS3AFileSystem(fs)) {
> nit: merge the branches?
Done
http://gerrit.cloudera.org:8080/#/c/13665/4/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@627
PS4, Line 627:
> s/filter? (FileStatusFilter isn't a thing anymore I guess.)
Done
--
To view, visit http://gerrit.cloudera.org:8080/13665
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa
Gerrit-Change-Number: 13665
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Fri, 21 Jun 2019 06:27:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4519/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 21 Jun 2019 06:18:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories The FileMetadataLoader is used to load the file information in when the table is loaded. By default, it lists all the files in the table/partition directory. Currently, it only skips the filenames which are invalid (hidden files and ones starting with "_" etc). However, it does not skip the directories which are temporary or hidden. In case of Hive when data is inserted into a table, it creates a temporary staging directory which is a hidden directory under the table location. When the insert in hive is completed, such staging directories are removed. But if there is a refresh called during that time, FileMetadataLoader will add the files in the staging directory as well. Not only this could cause temporary invalid results but it causes table to go in a bad state when these temporary directories are removed. The only work-around in such a case to issue a refresh on the table again. This patch adds logic in the filemetadataloader to ignore such temporary staging directories. Unfortunately, hadoop does not provide a API which can recursively list files in a directory and skip certain directories. This patch addes this logic of filtering into existing RecursingIterator in FileSystemUtil. Testing: 1. Added a new test in FilemetadataloaderTest and modified existing ones in AcidUtils 2. Ran concurrent inserts from Hive while issuing refresh in a loop on Impala side. Earlier this would cause the table to go into a bad state. Now, it works fine for the staging directories. It still runs into a FileNotFoundException from the impalad when there are insert overwrite statements in Hive Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java 3 files changed, 95 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13665/4 -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 4 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 21 Jun 2019 06:01:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. IMPALA-8652 Illegal delimiter error in shell has unknown error Problem: When assign --output_delimiter to invalid value, the validation of the argument is done only after the query is running, ValueError is raised in DelimitedOutputFormatter and caught in _exec_stmt in shell Solution: Add --output_delimiter option check before impala-shell initialization Remove delimiter length check in DelimitedOutputFormatter Testing: tests/shell/test_shell_commandline.py passed Example: $ impala-shell.sh -B --output_delimiter '||' -q 'select 1,1,1' Illegal delimiter ||, the delimiter must be a 1-character string. Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Reviewed-on: http://gerrit.cloudera.org:8080/13690 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M shell/impala_shell.py M shell/shell_output.py 2 files changed, 9 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 5 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12974 )
Change subject: IMPALA-7608: Estimate row count from file size when no stats
available
..
IMPALA-7608: Estimate row count from file size when no stats available
Added the feature that computes an estimated number of rows in the current
hdfs table if the statistics for the cardinality of the current hdfs table is
not
available.
Also added an additional query option to revert the change in case of
regression.
Testing:
(1) In CardinalityTest.java, replaced the original statement
"verifyCardinality("SELECT a FROM functional.tinytable", -1);" in
the method testBasicsWithoutStats() with
"verifyCardinality("SELECT a FROM functional.tinytable", 2);".
(2) In CarginalityTest.java, added more tests to check the cardinality
of most PlanNode implementations. For each tested PlanNode, the behaviors
before and after we disable the feature are both tested.
(3) In set.test, modified three related test cases to make sure that
the added query option is included after executing "set all" in various
scenarios.
(4) There are 8 JUnit tests in PlannerTest.java that would produce different
distributed query plans when this feature is enabled. Added an additional
JUnit test for 6 of those 8 affected JUnit tests when this feature is
enabled. Specifically, each tested query in a newly added test files involves
at least one hdfs table without available statistics.
We do not add test cases for 2 of the affected JUnit tests when this feature
is enabled since it results in flaky tests. These two JUnit tests are
testResourceRequirements() and testSpillableBufferSizing(). In this patch
we only test them when the feature is disabled.
(5) There are 5 Python end to end tests that consist of queries that would
produce different results. Added an additional query for each affected query
when this feature is disabled.
Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Reviewed-on: http://gerrit.cloudera.org:8080/12974
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsCompression.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
A
testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
A
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
A
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
23 files changed, 2,145 insertions(+), 25 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 25
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 24: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 24 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 21 Jun 2019 03:28:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7947: script to push images to docker repo
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13698 ) Change subject: IMPALA-7947: script to push images to docker repo .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3706/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0996b090f513351b58c801ed7149f80c4188f903 Gerrit-Change-Number: 13698 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:53:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7947: script to push images to docker repo
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13698 ) Change subject: IMPALA-7947: script to push images to docker repo .. Patch Set 1: Code-Review+1 (1 comment) Looks good to me. http://gerrit.cloudera.org:8080/#/c/13698/1/docker/push-images.sh File docker/push-images.sh: http://gerrit.cloudera.org:8080/#/c/13698/1/docker/push-images.sh@54 PS1, Line 54: required Nit: Typo -- To view, visit http://gerrit.cloudera.org:8080/13698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0996b090f513351b58c801ed7149f80c4188f903 Gerrit-Change-Number: 13698 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:44:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8425: part 2: avoid chown when building containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13699 ) Change subject: IMPALA-8425: part 2: avoid chown when building containers .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3707/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 Gerrit-Change-Number: 13699 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:41:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8425: part 2: avoid chown when building containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13699 ) Change subject: IMPALA-8425: part 2: avoid chown when building containers .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4517/ -- To view, visit http://gerrit.cloudera.org:8080/13699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 Gerrit-Change-Number: 13699 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:41:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8425: part 2: avoid chown when building containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13699 ) Change subject: IMPALA-8425: part 2: avoid chown when building containers .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4518/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 Gerrit-Change-Number: 13699 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:41:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8425: part 2: avoid chown when building containers
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13699 to look at the new patch set (#2). Change subject: IMPALA-8425: part 2: avoid chown when building containers .. IMPALA-8425: part 2: avoid chown when building containers This reduces the size of an image from 1.36GB to 705MB with a release build on my system. Thanks to Joe McDonnell for the suggestion. Testing: Precommit docker tests are sufficient to validate that the containers are functional. Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 --- M docker/impala_base/Dockerfile 1 file changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13699/2 -- To view, visit http://gerrit.cloudera.org:8080/13699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 Gerrit-Change-Number: 13699 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8425: part 2: avoid chown when building containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13699 ) Change subject: IMPALA-8425: part 2: avoid chown when building containers .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4517/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 Gerrit-Change-Number: 13699 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 21 Jun 2019 02:40:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8425: part 2: avoid chown when building containers
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13699 Change subject: IMPALA-8425: part 2: avoid chown when building containers .. IMPALA-8425: part 2: avoid chown when building containers This reduces the size of an image from 1.36GB to 705MB with a release build on my system. Testing: Precommit docker tests are sufficient to validate that the containers are functional. Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 --- M docker/impala_base/Dockerfile 1 file changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13699/1 -- To view, visit http://gerrit.cloudera.org:8080/13699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5476a97a7a030499a60a6cef67f8c3cdffa7e756 Gerrit-Change-Number: 13699 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. IMPALA-7802: Close connections of idle client sessions Previously, if idle session timeout is set either via startup flag or query options, a client session will expire after that set period of inactivity. However, the network connection and the service thread of an expired session will still be around until the session is closed by the client. This is highly undesirable as these idle sessions still count towards the quota bound by --fe_esrvice_threads, so if the total number of sessions (including the idle ones) reaches that upper bound, all incoming new session will block until some of the existing sessions exit. There is no time bound on when those expired sessions will be closed. In some sense, leaving many idle sessions opened is a denial-of-service attack on Impala. This change implements support for closing expired client sessions. In particular, a new flag --idle_client_poll_time_s is added to specify a time interval in seconds of client's inactivity which will cause an idle service thread of a client connection to wake up and check if all sessions associated with the connection are idle. If so, the connection will be closed. This allows the service threads to be freed up without waiting for client to close the connections. Testing done: - core build - new targeted test which verifies the connections of expired sessions are closed. - verified the flags function as expected in a secure cluster with Kerberos + SSL Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Reviewed-on: http://gerrit.cloudera.org:8080/13607 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/runtime/client-cache.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M tests/custom_cluster/test_hs2.py M tests/custom_cluster/test_session_expiration.py 11 files changed, 329 insertions(+), 94 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 21 Jun 2019 02:34:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7947: script to push images to docker repo
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13698 Change subject: IMPALA-7947: script to push images to docker repo .. IMPALA-7947: script to push images to docker repo docker/push-images.sh will push locally built images to a remote docker repo, prefixed with some string. See the script for details on usage. Testing: Manually tested pushing to dockerhub and to a private docker repository. Change-Id: I0996b090f513351b58c801ed7149f80c4188f903 --- A docker/push-images.sh 1 file changed, 62 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/13698/1 -- To view, visit http://gerrit.cloudera.org:8080/13698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0996b090f513351b58c801ed7149f80c4188f903 Gerrit-Change-Number: 13698 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 21 Jun 2019 01:22:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 11: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 21 Jun 2019 00:36:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 4 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 21 Jun 2019 00:30:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4516/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 4 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 21 Jun 2019 00:30:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 3: Code-Review+2 Thanks for the quick turnaround! -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 21 Jun 2019 00:30:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3705/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 23:27:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. IMPALA-7534. Handle invalidation races in CatalogdMetaProvider This handles a race condition in which a cache invalidation concurrent with a cache load would potentially be skipped, causing out-of-date data to persist in the cache. This would present itself as spurious "table not found" errors. A new test case triggers the issue reliably by injecting latency into the metadata fetch RPC and running DDLs concurrently on the same database across 8 threads. With the fix, the test passes reliably. Another option to fix this might have been to switch to Caffeine instead of Guava's loading cache. However, Caffeine requires Java 8, and LocalCatalog is being backported to Impala 2.x which still can run on Java 7. So, working around the Guava issue will make backporting (and future backports) easier. Change-Id: I70f377db88e204825a909389f28dc3451815235c Reviewed-on: http://gerrit.cloudera.org:8080/13664 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/catalog-op-executor.cc M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M tests/custom_cluster/test_local_catalog.py 4 files changed, 260 insertions(+), 22 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 20 Jun 2019 23:06:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Jiawei Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. IMPALA-8652 Illegal delimiter error in shell has unknown error Problem: When assign --output_delimiter to invalid value, the validation of the argument is done only after the query is running, ValueError is raised in DelimitedOutputFormatter and caught in _exec_stmt in shell Solution: Add --output_delimiter option check before impala-shell initialization Remove delimiter length check in DelimitedOutputFormatter Testing: tests/shell/test_shell_commandline.py passed Example: $ impala-shell.sh -B --output_delimiter '||' -q 'select 1,1,1' Illegal delimiter ||, the delimiter must be a 1-character string. Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 --- M shell/impala_shell.py M shell/shell_output.py 2 files changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13690/3 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. IMPALA-8443: Record time spent in authorization in the runtime profile The analysis and authorization is handled together as authorization depends on the results of the analysis. The timeline EventSequence is moved into the AuthorizationContext and the markEvent is called during the postAuthorize method. In some cases the EventSequence is not available when the AuthorizationContext is created therefore it is wrapped in Optional. Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Reviewed-on: http://gerrit.cloudera.org:8080/13353 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/EventSequence.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M tests/observability/test_log_fragments.py M tests/query_test/test_observability.py 13 files changed, 84 insertions(+), 27 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 12 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 22:42:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3704/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 22:26:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13690 )
Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error
..
Patch Set 2:
(2 comments)
A couple of minor nits, then looks good to go.
http://gerrit.cloudera.org:8080/#/c/13690/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13690/2//COMMIT_MSG@19
PS2, Line 19: tests/shell/test_shell_commandline.py passed
Can you include an example of what the new output looks like? Just to
illustrate that the issue is fixed.
http://gerrit.cloudera.org:8080/#/c/13690/2/shell/shell_output.py
File shell/shell_output.py:
http://gerrit.cloudera.org:8080/#/c/13690/2/shell/shell_output.py@54
PS2, Line 54: self.field_delim = field_delim.decode('string-escape')
I'd suggest adding an assert here, just to document and check the assumption
that the delimiter was validated elsewhere, i.e.
assert len(self.field_delim) != 1
--
To view, visit http://gerrit.cloudera.org:8080/13690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05
Gerrit-Change-Number: 13690
Gerrit-PatchSet: 2
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 20 Jun 2019 22:06:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 22:00:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 24: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 24 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 21:58:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 23: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 23 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 21:58:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 24: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4515/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 24 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 21:58:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13665 )
Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp
directories
..
Patch Set 2:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/13665/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13665/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-8663 : [WIP] FileMetadataLoader should skip listing files
in hidden and tmp directories
> nit: wrap
Done
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@549
PS3, Line 549:if (recursive) {
: // The Hadoop FileSystem API doesn't provide a recursive
listStatus call that
: // doesn't also fetch block locations, and fetching
block locations is expensive.
: // Here, our caller specifically doesn't need block
locations, so we don't want to
: // call the expensive 'listFiles' call on HDFS. Instead,
we need to "manually"
: // recursively call FileSystem.listStatusIterator().
: //
: // Note that this "manual" recursion is not actually any
slower than the recursion
: // provided by the HDFS 'listFiles(recursive=true)' API,
since the HDFS wire
: // protocol doesn't provide any such recursive support
anyway. In other words,
: // the API that looks like a single recursive call is
just as bad as what we're
: // doing here.
: //
: // However, S3 actually implements
'listFiles(recursive=true)' with a faster path
: // which natively recurses. In that case, it's quite
preferable to use 'listFiles'
: // even though it returns LocatedFileStatus objects with
"fake" blocks which we
: // will ignore.
: if (isS3AFileSystem(fs)) {
: return listFiles(fs, p, recursive);
: }
:
: return new RecursingIteratorWithFilter(fs, p, true);
: }
:
: return new RecursingIteratorWithFilter(fs, p, fal
> I think this is equivalent to...
yeah, makes sense. Changed.
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@581
PS3, Line 581: erface
> nit: used ?
Done
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@585
PS3, Line 585:
> nit: FILTER? (predicate is a java construct)
Done
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@642
PS3, Line 642: * certain files/directories. This is a implementation of
> This is always used, no? Hard code it?
I was hoping to make this class generic enough to take in any predicate. But
for now, probably makes sense to hard code.
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@643
PS3, Line 643:* {@link org.apache.hadoop.fs.RemoteIterator}
> Document this?
Done
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@658
PS3, Line 658: private final boolean useListStatus_;
> line too long (94 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
File fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java:
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@100
PS2, Line 100: dstFs.deleteOnExit(tmpTestPath);
> Are you sure?
The FileSystem#close is called when the JVM exits. See this,
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L1618
given this discussion, its probably clearer to use try with resources to make
it explicit to the reader. Made the change accordingly
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@112
PS2, Line 112: 24
> not sure I follow. I think what were asserting here is that the number of l
Isn't this approach assuming the load() method itself is not buggy? Eg, in the
suggested approach test will pass if load() method is hardcoded to return a
fixed number of files (my point being, a unrelated bug in FileMetadataLoader
will cause a false positive in the suggested approach)
In my opinion, comparing it to a known hard-coded number of files (or a trusted
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Jiawei Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. IMPALA-8652 Illegal delimiter error in shell has unknown error Problem: When assign --output_delimiter to invalid value, the validation of the argument is done only after the query is running, ValueError is raised in DelimitedOutputFormatter and caught in _exec_stmt in shell Solution: Add --output_delimiter option check before impala-shell initialization Remove delimiter length check in DelimitedOutputFormatter Testing: tests/shell/test_shell_commandline.py passed Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 --- M shell/impala_shell.py M shell/shell_output.py 2 files changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13690/2 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4514/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 20 Jun 2019 21:28:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 20 Jun 2019 21:28:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13689 ) Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Backport KUDU-2871 (part 1): disable TLS 1.3. Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Reviewed-on: http://gerrit.cloudera.org:8080/13689 Reviewed-by: Todd Lipcon Reviewed-by: Alexey Serbin Tested-by: Impala Public Jenkins --- M be/src/kudu/rpc/client_negotiation.cc M be/src/kudu/security/tls_context.cc 2 files changed, 10 insertions(+), 1 deletion(-) Approvals: Todd Lipcon: Looks good to me, approved Alexey Serbin: Looks good to me, but someone else must approve Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. IMPALA-8553,IMPALA-8552: fix checks for remote cluster Apparently IMPALA_REMOTE_URL is not generally used for remote cluster tests: only --testing_remote_cluster is reliably set. Fix the is_remote_cluster() implementation to take into account REMOTE_DATA_LOAD and --testing_remote_cluster in addition to IMPALA_REMOTE_URL. Consistently use is_remote_cluster() in other tests instead of checking the pytest flag directly. There were a few lifecycle headaches with how ImpalaTestClusterProperties is used: * common.environ is imported from conftest, which means that the top-level code in the file runs *before* pytest command-line arguments have been registered and parsed. * ImpalaTestClusterProperties is used by various code, like build_flavor_timeout(), which runs before pytest command-line arguments have been parsed. * ImpalaTestClusterProperties is called from non-pytest scripts like start-impala-cluster.py, so the command-line arguments are not available. I dealt with the above challenges by making a few changes to do the detection later: * Lazily initializing a singleton ImpalaTestClusterProperties. This was not strictly necessary but makes the whole problem less sensitive to import order and module dependencies. * Adding cluster_properties fixture to make ImpalaTestClusterProperties available in tests without additional boilerplate. * Removing the caching of the local/remote build calculation. ImpalaTestClusterProperties is instantiated outside of python tests, but is_remote_cluster() is only called from python tests, so if we check flags in is_remote_cluster() we'll get the right results reliably. As a workaround to unblock remote tests, also assume catalog_v1 if accessing the web UI fails. Testing: Ran core tests against a regular minicluster. Ran tests against a remote cluster Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Reviewed-on: http://gerrit.cloudera.org:8080/13386 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M tests/common/environ.py M tests/common/skip.py M tests/conftest.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_automatic_invalidation.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/hs2/test_hs2.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_kudu.py M tests/query_test/test_mt_dop.py M tests/query_test/test_spilling.py M tests/shell/test_shell_commandline.py M tests/shell/util.py M tests/webserver/test_web_pages.py 17 files changed, 144 insertions(+), 106 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 20:27:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py@1733 PS1, Line 1733: if options.output_delimiter != "\\t": Can we not check if the delimiter is not empty then perform the validation? http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py@1738 PS1, Line 1738: sys.exit(1) Raise FatalShellException to be consistent with the rest of the code here. Can we also add/update the test to catch this issue? -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 20:24:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13689 ) Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 20:24:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4513/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 19:53:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3703/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 19:48:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Jiawei Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13690 Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. IMPALA-8652 Illegal delimiter error in shell has unknown error Problem: When assign --output_delimiter to invalid value, the validation of the argument is done only after the query is running, ValueError is raised in DelimitedOutputFormatter and caught in _exec_stmt in shell Solution: Add --output_delimiter option check before impala-shell initialization Remove delimiter length check in DelimitedOutputFormatter Testing: tests/shell/test_shell_commandline.py passed Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 --- M shell/impala_shell.py M shell/shell_output.py 2 files changed, 8 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13690/1 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8665 Include extra info in error message when date cast fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13680 ) Change subject: IMPALA-8665 Include extra info in error message when date cast fails .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3702/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 Gerrit-Change-Number: 13680 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 20 Jun 2019 18:55:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip listing files in hidden and tmp directories
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13665 )
Change subject: IMPALA-8663 : FileMetadataLoader should skip listing files in
hidden and tmp directories
..
Patch Set 3:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/13665/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13665/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-8663 : FileMetadataLoader should skip listing files in
hidden and tmp directories
nit: wrap
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@549
PS3, Line 549:if (recursive) {
: // The Hadoop FileSystem API doesn't provide a recursive
listStatus call that
: // doesn't also fetch block locations, and fetching
block locations is expensive.
: // Here, our caller specifically doesn't need block
locations, so we don't want to
: // call the expensive 'listFiles' call on HDFS. Instead,
we need to "manually"
: // recursively call FileSystem.listStatusIterator().
: //
: // Note that this "manual" recursion is not actually any
slower than the recursion
: // provided by the HDFS 'listFiles(recursive=true)' API,
since the HDFS wire
: // protocol doesn't provide any such recursive support
anyway. In other words,
: // the API that looks like a single recursive call is
just as bad as what we're
: // doing here.
: //
: // However, S3 actually implements
'listFiles(recursive=true)' with a faster path
: // which natively recurses. In that case, it's quite
preferable to use 'listFiles'
: // even though it returns LocatedFileStatus objects with
"fake" blocks which we
: // will ignore.
: if (isS3AFileSystem(fs)) {
: return listFiles(fs, p, recursive);
: }
:
: return new RemoteIteratorWithFilter(fs, p, true);
: }
:
: return new RemoteIteratorWithFilter(fs, p, false)
I think this is equivalent to...
if (S3) return listFiles(fs, p, recursive);
return new RemoteIteratorWithFilter(fs, p, recursive);
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@581
PS3, Line 581: useful
nit: used ?
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@585
PS3, Line 585: PREDICATE
nit: FILTER? (predicate is a java construct)
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@642
PS3, Line 642: private final Predicate fileStatusPredicate_;
This is always used, no? Hard code it?
http://gerrit.cloudera.org:8080/#/c/13665/3/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@643
PS3, Line 643: private final boolean useListStatus_;
Document this?
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
File fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java:
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@100
PS2, Line 100: dstFs.deleteOnExit(tmpTestPath);
> don't think so. This will clean up the directory when the JVM exits. The ot
Are you sure?
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2505
Its done in close(), so you probably need to wrap it in try-with-resources or
explicitly call in a finally {}
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@112
PS2, Line 112: 24
> I am vary of using load() to test load() method's behavior. Seems like a an
not sure I follow. I think what were asserting here is that the number of
loaded files in the sourcePath would be the same with/without the filters. So
why not load the sourcePath, get the the numFiles, add the junk folders, load
again and compare the load numbers? I think thats better than hardcoding.
--
To view, visit http://gerrit.cloudera.org:8080/13665
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa
Gerrit-Change-Number: 13665
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Re
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3701/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 20 Jun 2019 18:38:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8665 Include extra info in error message when date cast fails
Jiawei Wang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13680 ) Change subject: IMPALA-8665 Include extra info in error message when date cast fails .. IMPALA-8665 Include extra info in error message when date cast fails When users cast string to date. It's hard for users to debug right now if users running millions of rows casting while casting failed without extra information. Solution: Add the failed data value into error message. Testing: changes -> date-partitioning.test & date.test query_test/test_date_queries.py test passed Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 --- M be/src/exprs/cast-functions-ir.cc M testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test M testdata/workloads/functional-query/queries/QueryTest/date.test 3 files changed, 12 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/13680/3 -- To view, visit http://gerrit.cloudera.org:8080/13680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 Gerrit-Change-Number: 13680 Gerrit-PatchSet: 3 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang
[Impala-ASF-CR] IMPALA-8665 Include extra info in error message when date cast fails
Jiawei Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/13680 )
Change subject: IMPALA-8665 Include extra info in error message when date cast
fails
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc
File be/src/exprs/cast-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc@310
PS2, Line 310: DateVal CastFunctions::CastToDateVal(FunctionContext* ctx, const
StringVal& val) {
> Please cover this change with tests. If you grep for "String to Date parse
I have already tested all the tests you mentioned. They all passed. The way
tests handle this is to find a string match in the output. So without changing
the old tests, this still passes.
Do you want me to change the tests cases? (i.e. add the additional information
for string match)
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc@314
PS2, Line 314: (char *)
> The preferred way of converting here is using reinterpret_cast as seen abov
Done
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc@316
PS2, Line 316: invalidVal
> Have you tried to simply pass the char* here? Is that a null terminated str
It's not working because StringVal.ptr is not a null-terminated string.
http://gerrit.cloudera.org:8080/#/c/13680/2/be/src/exprs/cast-functions-ir.cc@316
PS2, Line 316: c_str()
> Do you need to convert the return value of Substitute() to char*?
The reason why I used Substitute and .c_str() is that I am following the other
SetError() examples. The following is one of the example that use SetError()
function to pass a variable inside a string. To use SetError, char* is required.
agg_fn_ctx_->SetError(Substitute("UDA Serialize() and Finalize() must return
same pointer as input for $0 intermediate",
dst_slot_desc.type().DebugString()).c_str());
--
To view, visit http://gerrit.cloudera.org:8080/13680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04
Gerrit-Change-Number: 13680
Gerrit-PatchSet: 2
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Comment-Date: Thu, 20 Jun 2019 18:14:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12940 )
Change subject: IMPALA-7322: Add storage wait time to profile
..
Patch Set 11:
(3 comments)
lgtm, a few suggestions for cleanup, can +2 after that.
http://gerrit.cloudera.org:8080/#/c/12940/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/12940/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@945
PS11, Line 945:
: final Timer storageLdTimer =
:
getMetrics().getTimer(Table.STORAGE_METADATA_LOAD_DURATION_METRIC);
: final Clock clock = Clock.defaultClock();
: long startTime;
: storageMetadataLoadTime_ = 0;
:
: // Load partition and file metadata
: if (reuseMetadata) {
: // Incrementally update this table's partitions and
file metadata
: Preconditions.checkState(
: partitionsToUpdate == null ||
loadParitionFileMetadata);
:
: try {
: startTime = clock.getTick();
: updateMdFromHmsTable(msTbl);
: // Time spent on getting file system and check
access levels.
: storageMetadataLoadTime_ += clock.getTick() -
startTime;
: if (msTbl.getPartitionKeysSize() == 0) {
: startTime = clock.getTick();
: if (loadParitionFileMetadata)
updateUnpartitionedTableFileMd();
: storageMetadataLoadTime_ += clock.getTick() -
startTime;
: } else {
: long partitionfileLoadTime =
updatePartitionsFromHms(
: client, partitionsToUpdate,
loadParitionFileMetadata);
: storageMetadataLoadTime_ += partitionfileLoadTime;
: }
: } finally {
: storageLdTim
Can we clean this up a bit by making all the update/load methods return the
time spent in the loading, sum up all of them and finally update the metric?
I think that will be cleaner.
http://gerrit.cloudera.org:8080/#/c/12940/11/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/12940/11/fe/src/main/java/org/apache/impala/catalog/Table.java@118
PS11, Line 118: .
Mention that it is updated on every reload of the table?
http://gerrit.cloudera.org:8080/#/c/12940/11/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/12940/11/tests/query_test/test_observability.py@572
PS11, Line 572: def test_query_profile_storage_load_time1(self):
: """Test that when a query needs load metadata for table(s),
the
: storage load time should be in the profile. Tests file
systems."""
: self.__check_query_profile_storage_load_time("functional")
:
: @SkipIfS3.hbase
: @SkipIfLocal.hbase
: @SkipIfIsilon.hbase
: @SkipIfABFS.hbase
: @SkipIfADLS.hbase
: def test_query_profile_storage_load_time2(self):
: """Test that when a query needs load metadata for table(s),
the
: storage load time should be in the profile. Tests kudu and
hbase."""
: # KUDU table
:
self.__check_query_profile_storage_load_time("functional_kudu")
:
: # HBASE table
:
self.__check_query_profile_storage_load_time("functional_hbase")
you could just do something like
for suffix in ["", "_kudu", "_hbase"]:
check_query_...("functional" + siffix)
Please avoid test names that are hard to follow.
Btw, why are we skipping hbase?
--
To view, visit http://gerrit.cloudera.org:8080/12940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660
Gerrit-Change-Number: 12940
Gerrit-PatchSet: 11
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Thu, 20 Jun 2019 18:13:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 4: Code-Review+2 Carry Thomas' +2 -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 20 Jun 2019 18:02:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13607/2/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/13607/2/be/src/service/impala-server.cc@219 PS2, Line 219: DEFINE_int32(idle_client_poll_period_ms, 3, "The poll period, in milliseconds, after " > Maybe make this seconds not ms? I don't think there's a reason anyone would Done -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 20 Jun 2019 17:58:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Michael Ho has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. IMPALA-7802: Close connections of idle client sessions Previously, if idle session timeout is set either via startup flag or query options, a client session will expire after that set period of inactivity. However, the network connection and the service thread of an expired session will still be around until the session is closed by the client. This is highly undesirable as these idle sessions still count towards the quota bound by --fe_esrvice_threads, so if the total number of sessions (including the idle ones) reaches that upper bound, all incoming new session will block until some of the existing sessions exit. There is no time bound on when those expired sessions will be closed. In some sense, leaving many idle sessions opened is a denial-of-service attack on Impala. This change implements support for closing expired client sessions. In particular, a new flag --idle_client_poll_time_s is added to specify a time interval in seconds of client's inactivity which will cause an idle service thread of a client connection to wake up and check if all sessions associated with the connection are idle. If so, the connection will be closed. This allows the service threads to be freed up without waiting for client to close the connections. Testing done: - core build - new targeted test which verifies the connections of expired sessions are closed. - verified the flags function as expected in a secure cluster with Kerberos + SSL Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/runtime/client-cache.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M tests/custom_cluster/test_hs2.py M tests/custom_cluster/test_session_expiration.py 11 files changed, 329 insertions(+), 94 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13607/4 -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3700/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 17:54:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3699/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 17:42:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 11: Code-Review+1 Any rebase conflicts that need to be reviewed? Otherwise, carrying +1 -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 11 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 20 Jun 2019 17:42:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4512/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 20 Jun 2019 17:40:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 20 Jun 2019 17:40:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 11: Yes, I have seen that, just in case I have started a dry run :). Thanks for re-running the merge. -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:15:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 275 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/5 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py File tests/authorization/test_authorized_proxy.py: http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@195 PS4, Line 195: s > what are these commented-out lines? I meant to delete that. Done. http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@231 PS4, Line 231: > typo? Done -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 17:14:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4511/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:14:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:14:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 10: > Patch Set 10: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4500/ This looks like some flaky Maven issue. I'm re-running the merge. 23:59:12 ] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project query-event-hook-api: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.3 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-compiler-plugin:jar:3.3 -> org.codehaus.plexus:plexus-container-default:jar:1.5.5 -> org.apache.xbean:xbean-reflect:jar:3.4: Failed to read artifact descriptor for org.apache.xbean:xbean-reflect:jar:3.4: Could not transfer artifact org.apache.xbean:xbean-reflect:pom:3.4 from/to central (https://repo.maven.apache.org/maven2): /home/ubuntu/.m2/repository/org/apache/xbean/xbean-reflect/3.4/xbean-reflect-3.4.pom.part (No such file or directory) -> [Help 1] 23:59:12 ] [ERROR] 23:59:12 ] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 23:59:12 ] [ERROR] Re-run Maven using the -X switch to enable full debug logging. 23:59:12 ] [ERROR] 23:59:12 ] [ERROR] For more information about the errors and possible solutions, please read the following articles: 23:59:12 ] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException 23:59:12 ] mvn -B install -DskipTests exited with code 0 23:59:12 ] query-event-hook-api/CMakeFiles/query-event-hook-api.dir/build.make:57: recipe for target 'query-event-hook-api/CMakeFiles/query-event-hook-api' failed 23:59:12 ] make[3]: *** [query-event-hook-api/CMakeFiles/query-event-hook-api] Error 1 23:59:12 ] CMakeFiles/Makefile2:14640: recipe for target 'query-event-hook-api/CMakeFiles/query-event-hook-api.dir/all' failed 23:59:12 ] make[2]: *** [query-event-hook-api/CMakeFiles/query-event-hook-api.dir/all] Error 2 23:59:12 ] make[2]: *** Waiting for unfinished jobs -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:13:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:08:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 9: (4 comments) Thanks for the review http://gerrit.cloudera.org:8080/#/c/13545/9/be/src/scheduling/scheduler-test-util.cc File be/src/scheduling/scheduler-test-util.cc: http://gerrit.cloudera.org:8080/#/c/13545/9/be/src/scheduling/scheduler-test-util.cc@47 PS9, Line 47: > nit: double space Done http://gerrit.cloudera.org:8080/#/c/13545/9/be/src/scheduling/scheduler-test.cc File be/src/scheduling/scheduler-test.cc: http://gerrit.cloudera.org:8080/#/c/13545/9/be/src/scheduling/scheduler-test.cc@18 PS9, Line 18: #include > Is this one needed? Oh, good point, this is leftover from an approach I ended up not using. Removed. http://gerrit.cloudera.org:8080/#/c/13545/9/be/src/scheduling/scheduler-test.cc@228 PS9, Line 228: cluster.AddHosts(num_data_nodes, false, true); > nit: use AddRemoteCluster()? Switched this over. Documented that CreateRemoteCluster() puts the Impala nodes first, so the indices for the AddSingleBlockTable() call needed to change. http://gerrit.cloudera.org:8080/#/c/13545/9/be/src/scheduling/scheduler-test.cc@376 PS9, Line 376: cluster.AddHosts(num_impala_nodes, true, false); > nit: AddRemoteCluster()? Good catch, these statements shouldn't be here, as this uses CreateRemoteCluster(). -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 17:02:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Hello Lars Volker, Tim Armstrong, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13545 to look at the new patch set (#10). Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. IMPALA-8630: Hash the full path when calculating consistent remote placement Consistent remote placement currently uses the relative filename within a partition for the consistent hash. If the relative filenames for different partitions have a simple naming scheme, then multiple partitions may have files of the same name. This is true for some tables written by Hive (e.g. in our minicluster the tpcds.store_sales has this problem). This can lead to unbalanced placement of remote ranges. This adds a partition_path_hash to the THdfsFileSplit and THdfsFileSplitGeneratorSpec, calculated in the frontend (which has all of the partition information). The scheduler hashes this in addition to the relative path. Testing: - Added several new scheduler tests that verify the consistent remote scheduling sees blocks with different relative paths, partition paths, or offsets as distinct. - Ran core tests Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b --- M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java 7 files changed, 332 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13545/10 -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13648 ) Change subject: IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3698/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If404bffdaf055c769e79ffa8f193bac415cfdd1a Gerrit-Change-Number: 13648 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 20 Jun 2019 17:01:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 20 Jun 2019 16:57:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3697/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 23 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 16:39:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py File tests/authorization/test_authorized_proxy.py: http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@195 PS4, Line 195: # what are these commented-out lines? http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@231 PS4, Line 231: # Try to user we are not authorized to delegate to. typo? -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 16:21:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE
Attila Jeges has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13648 ) Change subject: IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE .. IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE This change implements INTERVAL expression support for DATE type and adds several DATE related built-in functions. The efficiency of the DateValue::ToYearMonthDay() function used in many of the built-in functions below was also improved. The following functions are supported in Hive: INT YEAR(DATE d) Extracts year of the 'd' date, returns it as an int in 0- range. INT MONTH(DATE d) Extracts month of the 'd' date and returns it as an int in 1-12 range. INT DAY(DATE d), INT DAYOFMONTH(DATE d) Extracts day-of-month of the 'd' date and returns it as an int in 1-31 range. INT QUARTER(DATE d) Extracts quarter of the 'd' date and returns it as an int in 1-4 range. INT DAYOFWEEK(DATE d) Extracts day-of-week of the 'd' date and returns it as an int in 1-7 range. 1 is Sunday and 7 is Saturday. INT DAYOFYEAR(DATE d) Extracts day-of-year of the 'd' date and returns it as an int in 1-366 range. INT WEEKOFYEAR(DATE d) Extracts week-of-year of the 'd' date and returns it as an int in 1-53 range. STRING DAYNAME(DATE d) Returns the day field from a 'd' date, converted to the string corresponding to that day name. The range of return values is "Sunday" to "Saturday". STRING MONTHNAME(DATE d) Returns the month field from a 'd' date, converted to the string corresponding to that month name. The range of return values is "January" to "December". DATE NEXT_DAY(DATE d, STRING weekday) Returns the first date which is later than 'd' and named as 'weekday'. 'weekday' is 3 letters or full name of the day of the week. DATE LAST_DAY(DATE d) Returns the last day of the month which the 'd' date belongs to. INT DATEDIFF(DATE d1, DATE d2) Returns the number of days from 'd1' date to 'd2' date. DATE CURRENT_DATE() Returns the current date (in the local time zone). INT INT_MONTHS_BETWEEN(DATE d1, DATE d2) Returns the number of months between 'd1' and 'd2' dates, as an int representing only the full months that passed. If 'd1' represents an earlier date than 'd2', the result is negative. DOUBLE MONTHS_BETWEEN(DATE d1, DATE d2) Returns the number of months between 'd1' and 'd2' dates. Can include a fractional part representing extra days in addition to the full months between the dates. The fractional component is computed by dividing the difference in days by 31 (regardless of the month). If 'd1' represents an earlier date than 'd2', the result is negative. DATE ADD_YEARS(DATE d, INT/BIGINT num_years), DATE SUB_YEARS(DATE d, INT/BIGINT num_years) Adds/subtracts a specified number of years to a 'd' date value. DATE ADD_MONTHS(DATE d, INT/BIGINT num_months), DATE SUB_MONTHS(DATE d, INT/BIGINT num_months) Adds/subtracts a specified number of months to a date value. If 'd' is the last day of a month, the returned date will fall on the last day of the target month too. DATE ADD_DAYS(DATE d, INT/BIGINT num_days), DATE SUB_DAYS(DATE d, INT/BIGINT num_days) Adds/subtracts a specified number of days to a date value. DATE ADD_WEEKS(DATE d, INT/BIGINT num_weeks), DATE SUB_WEEKS(DATE d, INT/BIGINT num_weeks) Adds/subtracts a specified number of weeks to a date value. The following function doesn't exist in Hive but supported by Amazon Redshift INT DATE_CMP(DATE d1, DATE d2) Compares 'd1' and 'd2' dates. Returns: 1. NULL, if either 'd1' or 'd2' is NULL 2. -1 if d1 < d2 3. 1 if d1 > d2 4. 0 if d1 == d2 (https://docs.aws.amazon.com/redshift/latest/dg/r_DATE_CMP.html) Change-Id: If404bffdaf055c769e79ffa8f193bac415cfdd1a --- M be/src/benchmarks/date-benchmark.cc M be/src/codegen/impala-ir.cc M be/src/exprs/CMakeLists.txt A be/src/exprs/date-functions-ir.cc A be/src/exprs/date-functions.h M be/src/exprs/expr-test.cc M be/src/exprs/scalar-expr-evaluator.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc M be/src/exprs/timestamp-functions.h M be/src/exprs/udf-builtins.cc M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h M common/function-registry/impala_functions.py M fe/src/main/java/org/apache/impala/analysis/TimestampArithmeticExpr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java 17 files changed, 1,695 insertions(+), 178 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13648/4 -- To view, visit http://gerrit.cloudera.org:8080/13648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If404bffdaf055c769e79ffa8f193bac415cf
[Impala-ASF-CR] IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/13648 ) Change subject: IMPALA-7369: part 2: Add INTERVAL expr support and built-in functions for DATE .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/13648/3/be/src/benchmarks/date-benchmark.cc File be/src/benchmarks/date-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/13648/3/be/src/benchmarks/date-benchmark.cc@37 PS3, Line 37: 4.92X > Nice results! Can you mention this optimization in the commit message? Done http://gerrit.cloudera.org:8080/#/c/13648/3/be/src/benchmarks/date-benchmark.cc@169 PS3, Line 169: data.AddRange(DateValue(1965, 1, 1), DateValue(2020, 12, 31)); > It is possible that the sorted test data helps the current solution by maki Done. Somehow the results are even better with shuffled test data. http://gerrit.cloudera.org:8080/#/c/13648/3/be/src/runtime/date-value.cc File be/src/runtime/date-value.cc: http://gerrit.cloudera.org:8080/#/c/13648/3/be/src/runtime/date-value.cc@199 PS3, Line 199: int m = int(days_since_jan1 / 30.5); > Idea for +1 DCHECK: DCHECK(month_ranges[m] > days_since_jan1) Actually it should be the other way around: DCHECK(month_ranges[m] <= days_since_jan1) -- To view, visit http://gerrit.cloudera.org:8080/13648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If404bffdaf055c769e79ffa8f193bac415cfdd1a Gerrit-Change-Number: 13648 Gerrit-PatchSet: 3 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 20 Jun 2019 16:18:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Fang-Yu Rao has uploaded a new patch set (#23). (
http://gerrit.cloudera.org:8080/12974 )
Change subject: IMPALA-7608: Estimate row count from file size when no stats
available
..
IMPALA-7608: Estimate row count from file size when no stats available
Added the feature that computes an estimated number of rows in the current
hdfs table if the statistics for the cardinality of the current hdfs table is
not
available.
Also added an additional query option to revert the change in case of
regression.
Testing:
(1) In CardinalityTest.java, replaced the original statement
"verifyCardinality("SELECT a FROM functional.tinytable", -1);" in
the method testBasicsWithoutStats() with
"verifyCardinality("SELECT a FROM functional.tinytable", 2);".
(2) In CarginalityTest.java, added more tests to check the cardinality
of most PlanNode implementations. For each tested PlanNode, the behaviors
before and after we disable the feature are both tested.
(3) In set.test, modified three related test cases to make sure that
the added query option is included after executing "set all" in various
scenarios.
(4) There are 8 JUnit tests in PlannerTest.java that would produce different
distributed query plans when this feature is enabled. Added an additional
JUnit test for 6 of those 8 affected JUnit tests when this feature is
enabled. Specifically, each tested query in a newly added test files involves
at least one hdfs table without available statistics.
We do not add test cases for 2 of the affected JUnit tests when this feature
is enabled since it results in flaky tests. These two JUnit tests are
testResourceRequirements() and testSpillableBufferSizing(). In this patch
we only test them when the feature is disabled.
(5) There are 5 Python end to end tests that consist of queries that would
produce different results. Added an additional query for each affected query
when this feature is disabled.
Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsCompression.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
A
testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
A
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
A
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
23 files changed, 2,145 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/12974/23
--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 23
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13689 ) Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 15:47:47 + Gerrit-HasComments: No
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13689 ) Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3696/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 15:46:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13689 ) Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 15:44:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3695/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 15:36:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8686: docker entrypoint script execs daemon
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13682 ) Change subject: IMPALA-8686: docker entrypoint script execs daemon .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3694/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13682 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifefbe0a926cf9cfb8acbd37c3f691dc28847dd8b Gerrit-Change-Number: 13682 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 20 Jun 2019 15:35:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3693/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 11 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 20 Jun 2019 15:31:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13559 ) Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3692/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 20 Jun 2019 15:23:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13689 ) Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4510/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 20 Jun 2019 15:02:28 + Gerrit-HasComments: No
[Impala-ASF-CR] Backport KUDU-2871 (part 1): disable TLS 1.3.
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13689 Change subject: Backport KUDU-2871 (part 1): disable TLS 1.3. .. Backport KUDU-2871 (part 1): disable TLS 1.3. Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da --- M be/src/kudu/rpc/client_negotiation.cc M be/src/kudu/security/tls_context.cc 2 files changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/13689/1 -- To view, visit http://gerrit.cloudera.org:8080/13689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iae77e06906e01d8442e0f767e7f920bd330cc5da Gerrit-Change-Number: 13689 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4509/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 14:58:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/13386/13/tests/common/environ.py File tests/common/environ.py: http://gerrit.cloudera.org:8080/#/c/13386/13/tests/common/environ.py@328 PS13, Line 328: i flake8: F821 undefined name 'impala_url' -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 14:58:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 13: Code-Review+2 rebase conflict with my own change -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 14:58:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Hello David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13386 to look at the new patch set (#13). Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. IMPALA-8553,IMPALA-8552: fix checks for remote cluster Apparently IMPALA_REMOTE_URL is not generally used for remote cluster tests: only --testing_remote_cluster is reliably set. Fix the is_remote_cluster() implementation to take into account REMOTE_DATA_LOAD and --testing_remote_cluster in addition to IMPALA_REMOTE_URL. Consistently use is_remote_cluster() in other tests instead of checking the pytest flag directly. There were a few lifecycle headaches with how ImpalaTestClusterProperties is used: * common.environ is imported from conftest, which means that the top-level code in the file runs *before* pytest command-line arguments have been registered and parsed. * ImpalaTestClusterProperties is used by various code, like build_flavor_timeout(), which runs before pytest command-line arguments have been parsed. * ImpalaTestClusterProperties is called from non-pytest scripts like start-impala-cluster.py, so the command-line arguments are not available. I dealt with the above challenges by making a few changes to do the detection later: * Lazily initializing a singleton ImpalaTestClusterProperties. This was not strictly necessary but makes the whole problem less sensitive to import order and module dependencies. * Adding cluster_properties fixture to make ImpalaTestClusterProperties available in tests without additional boilerplate. * Removing the caching of the local/remote build calculation. ImpalaTestClusterProperties is instantiated outside of python tests, but is_remote_cluster() is only called from python tests, so if we check flags in is_remote_cluster() we'll get the right results reliably. As a workaround to unblock remote tests, also assume catalog_v1 if accessing the web UI fails. Testing: Ran core tests against a regular minicluster. Ran tests against a remote cluster Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 --- M tests/common/environ.py M tests/common/skip.py M tests/conftest.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_automatic_invalidation.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/hs2/test_hs2.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_kudu.py M tests/query_test/test_mt_dop.py M tests/query_test/test_spilling.py M tests/shell/test_shell_commandline.py M tests/shell/util.py M tests/webserver/test_web_pages.py 17 files changed, 144 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13386/13 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8686: docker entrypoint script execs daemon
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13682 Change subject: IMPALA-8686: docker entrypoint script execs daemon .. IMPALA-8686: docker entrypoint script execs daemon The script now execs the subprocess, which is required for signals, etc to be handled correctly. Change-Id: Ifefbe0a926cf9cfb8acbd37c3f691dc28847dd8b --- M docker/daemon_entrypoint.sh 1 file changed, 1 insertion(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/13682/2 -- To view, visit http://gerrit.cloudera.org:8080/13682 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifefbe0a926cf9cfb8acbd37c3f691dc28847dd8b Gerrit-Change-Number: 13682 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 11: Submit patch 11 for the rebase, thanks for the review. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 11 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 20 Jun 2019 14:51:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Anurag Mantripragada, Vihang Karajgaonkar, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12940 to look at the new patch set (#11). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: Ran queries that can trigger all of, none of or some of the related tables loading. Check query profile for each query. Check catalog metrics for each table. Add unit tests to test_observability.py Ran all core tests. Sample output: Profile:(storage-load-time is the added property): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/query_test/test_observability.py 7 files changed, 131 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/12940/11 -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 11 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/13559 )
Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables
..
Patch Set 6:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/13559/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13559/5//COMMIT_MSG@36
PS5, Line 36: TODO in following commits:
> It is not clear to me whether dynamic partitioning is supported or not.
Yeah, it is supported. Added tests about it.
http://gerrit.cloudera.org:8080/#/c/13559/5/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13559/5/be/src/exec/hdfs-table-sink.cc@214
PS5, Line 214: // Create final_hdfs_file_name_prefix and
tmp_hdfs_file_name_prefix.
> Can you mention the transactional naming logic in the comment?
Added comment.
http://gerrit.cloudera.org:8080/#/c/13559/5/be/src/exec/hdfs-table-sink.cc@696
PS5, Line 696: closed_ = true;
: }
:
> nit: can you improve readability? e.g if (IsTransactional()) return true; c
Done
http://gerrit.cloudera.org:8080/#/c/13559/5/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/13559/5/be/src/service/client-request-state.cc@817
PS5, Line 817: RETURN_IF_ERROR(GetCoordinator()->Wait());
: if (exec_request().__isset.transaction_id) {
:
RETURN_IF_ERROR(frontend_->CommitTransaction(exec_request().transaction_id));
: in_transaction_ = false;
: }
> I am not sure here, but the order of UpdateCatalog() and CommitTransaction(
UpdateCatalog() won't load the new write ids and won't see the new files until
they are not committed.
The problematic scenario is when we commit the transaction successfully but
UpdateCatalog() fails to update HMS.
http://gerrit.cloudera.org:8080/#/c/13559/5/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/13559/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1259
PS5, Line 1259: try {
> I think it would be more readable if the logic inside the try block would b
This 'doCreateExecRequest()' method also exist for the same reason. I'm not
sure about the usefulness of adding another method.
http://gerrit.cloudera.org:8080/#/c/13559/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1370
PS5, Line 1370: Exception e) {
> Should this only catch ImpalaException? e.g. if there is a null pointer exc
Done
http://gerrit.cloudera.org:8080/#/c/13559/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1372
PS5, Line 1372: try {
> This can also throw an exception, for example if HMS was shut down, and thi
For the general case maybe a composite exception type is the solution.
Here I think it's best to handle (log) the transaction exception and throw the
original one coming from analysis/planning.
http://gerrit.cloudera.org:8080/#/c/13559/5/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/13559/5/testdata/datasets/functional/functional_schema_template.sql@2611
PS5, Line 2611:
> nit: extra ;
Done
http://gerrit.cloudera.org:8080/#/c/13559/5/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test
File testdata/workloads/functional-query/queries/QueryTest/acid-insert.test:
http://gerrit.cloudera.org:8080/#/c/13559/5/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test@7
PS5, Line 7: QUERY
> It would be great to also read the tables back with HMS.
I have some interop tests at test_hms_integration.py
http://gerrit.cloudera.org:8080/#/c/13559/5/tests/metadata/test_hms_integration.py
File tests/metadata/test_hms_integration.py:
http://gerrit.cloudera.org:8080/#/c/13559/5/tests/metadata/test_hms_integration.py@662
PS5, Line 662: acid
> As this test runs in a unique_database, I don't think that it needs to run
Thanks for the tip!
http://gerrit.cloudera.org:8080/#/c/13559/5/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:
http://gerrit.cloudera.org:8080/#/c/13559/5/tests/query_test/test_insert.py@142
PS5, Line 142: def test_acid_insert(self, vector):
> What is the reason for not using a unique database?
The tables being used are tied to these tests only. They don't even contain any
data initially. I followed the pattern of test_insert().
http://gerrit.cloudera.org:8080/#/c/13559/5/tests/query_test/test_insert.py@146
PS5, Line 146: # We need to turn off capab
> @SkipIfHive2.acid already ensures that HIVE_MAJOR_VERSION >= 3.
Done
http://gerrit.cloudera.org:8080/#/c/13559/5/tests/query_test/test_insert.py@147
PS5, Line 147: # this python client doesn't have the capability for
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13559 ) Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/13559/6/tests/query_test/test_insert.py File tests/query_test/test_insert.py: http://gerrit.cloudera.org:8080/#/c/13559/6/tests/query_test/test_insert.py@26 PS6, Line 26: from tests.common.environ import HIVE_MAJOR_VERSION flake8: F401 'tests.common.environ.HIVE_MAJOR_VERSION' imported but unused http://gerrit.cloudera.org:8080/#/c/13559/6/tests/query_test/test_insert.py@155 PS6, Line 155: # flake8: E116 unexpected indentation (comment) -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 20 Jun 2019 14:43:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13559 to look at the new patch set (#6). Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. IMPALA-8636: Implement INSERT for insert-only ACID tables This commit adds INSERT support for insert-only ACID tables. The Frontend opens a transaction for queries that refer to transactional tables. For INSERT statements that write insert-only ACID tables it also allocates a write ID. The Frontend aborts the transaction if an error occurs during analysis/planning. The Backend gets the transaction id in TExecRequestState and the write id is set for the HDFS table sinks. The sinks write the files at their final destination which is an ACID base/delta directory. There is no need for finalization of transactional INSERTS. ClientRequestState commits the transaction in WaitInternal() if everything went well. If the transaction is still open in Done(), it means there was an error, therefore the transaction needs to be aborted. The Backend commits/aborts the transaction by calling the Frontend via JNI. Testing: * added new tables during dataload * added acid-insert.test file with INSERT statements against the new tables * added integration test with Hive to test_hms_integration.py. The test inserts data with Impala and reads with Hive. (These integration tests only run with exhaustive exploration strategy) TODO in following commits: * add locks and heartbeats * implement TRUNCATE (maybe in another commit) * CTAS creates files in the 'root' directory of the table/partition. It is handled correctly during SELECT, but would be better to create a base directory from the beginning. Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/coordinator.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.h M common/thrift/DataSinks.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/common/TransactionException.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/acid-insert.test M tests/metadata/test_hms_integration.py M tests/query_test/test_insert.py 25 files changed, 714 insertions(+), 119 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/13559/6 -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 12:02:32 + Gerrit-HasComments: No
