[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 16: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 16 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 19 Jun 2019 05:58:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. Add 3 configuration parameters to Admission Controller that scale with the number of hosts in the resource pool. These parameters are specified to the Impalad through the -llama_site_path flag which points to a Llama XML configuration file. The new configuration parameters are: + Max Running Queries Multiple - this floating point number is multiplied by the current total number of executors at runtime to give the maximum number of concurrently running queries allowed in the pool. This calculation is rounded up to the nearest integer so the result will always be at least one as long as the parameter is non-zero. + Max Queued Queries Multiple - this floating point number is multiplied by the current total number of executors at runtime to give the maximum number of queries that can be queued in the pool. This calculation is rounded up to the nearest integer so the result will always be at least one as long as the parameter is non-zero. + Max Memory Multiple - this number of bytes is multiplied by the current total number of executors at runtime to give the maximum memory available across the cluster for the pool. If any of these parameters have zero value then they will be ignored. In this case the corresponding non-scalable parameters will be used, if they are set. The new parameters are exposed through the webui. At various points in the code Admission Controller looks at the Pool Config objects to find non-scalable parameters such as the max number of queries that can run in the pool. These access have been encapsulated in functions that return the scalable version of the configuration value if the new scalable parameters are being used. Diagnostic messages are enhanced to show the origin of the encapsulated parameters. TESTING All end-to-end tests are running clean with ASAN. The unit test admission-controller-test.cc has been expanded to test the newly added code. Added an end-to-end test that adds and removes Impalads from a minicluster. Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Reviewed-on: http://gerrit.cloudera.org:8080/13307 Reviewed-by: Andrew Sherman Tested-by: Impala Public Jenkins --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_metrics.py M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/util/RequestPoolService.java M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java M fe/src/test/resources/fair-scheduler-test.xml M fe/src/test/resources/fair-scheduler-test2.xml M fe/src/test/resources/llama-site-test.xml M fe/src/test/resources/llama-site-test2.xml M testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M tests/common/impala_cluster.py M tests/custom_cluster/test_admission_controller.py M tests/webserver/test_web_pages.py M www/admission_controller.tmpl 20 files changed, 1,395 insertions(+), 302 deletions(-) Approvals: Andrew Sherman: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 17 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 05:17:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 04:43:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. IMPALA-8617: Add support for lz4 in parquet A new enum value LZ4_BLOCKED was added to the THdfsCompression enum, to distinguish it from the existing LZ4 codec. LZ4_BLOCKED codec represents the block compression scheme used by Hadoop. Its similar to SNAPPY_BLOCKED as far as the block format is concerned, with the only difference being the codec used for compression and decompression. Added Lz4BlockCompressor and Lz4BlockDecompressor classes for compressing and decompressing parquet data using Hadoop's lz4 block compression scheme. The Lz4BlockCompressor treats the input as a single block and generates a compressed block with following layout <4 byte big endian uncompressed size> <4 byte big endian compressed size> The hdfs parquet table writer should call the Lz4BlockCompressor using the ideal input size (unit of compression in parquet is a page), and so the Lz4BlockCompressor does not further break down the input into smaller blocks. The Lz4BlockDecompressor on the other hand should be compatible with blocks written by Impala and other engines in Hadoop ecosystem. It can decompress compressed data in following format <4 byte big endian uncompressed size> <4 byte big endian compressed size> ... <4 byte big endian compressed size> ... Externally users can now set the lz4 codec for parquet using: set COMPRESSION_CODEC=lz4 This gets translated into LZ4_BLOCKED codec for the HdfsParquetTableWriter. Similarly, when reading lz4 compressed parquet data, the LZ4_BLOCKED codec is used. Testing: - Added unit tests for LZ4_BLOCKED in decompress-test.cc - Added unit tests for Hadoop compatibility in decompress-test.cc, basically being able to decompress an outer block with multiple inner blocks (the Lz4BlockDecompressor description above) - Added interoperability tests for Hive and Impala for all parquet codecs. New test added to tests/custom_cluster/test_hive_parquet_codec_interop.py Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Reviewed-on: http://gerrit.cloudera.org:8080/13582 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/service/query-options-test.cc M be/src/util/codec.cc M be/src/util/compress.cc M be/src/util/compress.h M be/src/util/decompress-test.cc M be/src/util/decompress.cc M be/src/util/decompress.h M common/thrift/CatalogObjects.thrift M common/thrift/generate_error_codes.py M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/common/test_dimensions.py A tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/query_test/test_insert.py M tests/query_test/test_insert_parquet.py 18 files changed, 440 insertions(+), 59 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4493/ -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 03:06:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Reviewed-on: http://gerrit.cloudera.org:8080/13601 Reviewed-by: Todd Lipcon Tested-by: Impala Public Jenkins --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 22 files changed, 223 insertions(+), 144 deletions(-) Approvals: Todd Lipcon: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 02:30:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 21 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 02:26:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3675/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 19 Jun 2019 02:20:40 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8669 Support giving a terminal commit in compare branches.py
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13660 ) Change subject: IMPALA-8669 Support giving a terminal commit in compare_branches.py .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I7a4dfae26f8f616c6f550114edfbafd6fc1c068d Gerrit-Change-Number: 13660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 19 Jun 2019 01:42:42 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8669 Support giving a terminal commit in compare branches.py
Quanlong Huang has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/13660 )
Change subject: IMPALA-8669 Support giving a terminal commit in
compare_branches.py
..
IMPALA-8669 Support giving a terminal commit in compare_branches.py
Currently, bin/compare_branches.py (used in cherrypick-2.x-and-test
Jenkins job) will cherry-pick as more commits as possible. However, a
clean pick from the master branch does not mean it can always pass the
tests. If we find such a problematic commit, we'd like to let the
cherrypick-2.x-and-test Jenkins job cherry-pick till the commit before
it. And then submit only a patch for the problematic commit for review.
This patch adds a new parameter "source_terminal_commit" for specifying
where to stop. It's the full commit hash of the terminal commit in the
source branch. bin/compare_branches.py will stop after cherry-picking
this commit.
Tests:
- Test locally by pushing the 2.x branch to my repo ("self") and run
bin/compare_branches.py --cherry_pick --partial_ok \
--source_remote_name asf-gerrit --target_remote_name self -v \
--source_terminal_commit def5c881b1dee93848daea52d93a0602e11338ca
Change-Id: I7a4dfae26f8f616c6f550114edfbafd6fc1c068d
Reviewed-on: http://gerrit.cloudera.org:8080/13660
Reviewed-by: Tim Armstrong
Tested-by: Quanlong Huang
---
M bin/compare_branches.py
1 file changed, 9 insertions(+), 2 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved
Quanlong Huang: Verified
--
To view, visit http://gerrit.cloudera.org:8080/13660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I7a4dfae26f8f616c6f550114edfbafd6fc1c068d
Gerrit-Change-Number: 13660
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8663 : [WIP] FileMetadataLoader should skip listing files in hidden and tmp directories
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13665 )
Change subject: IMPALA-8663 : [WIP] FileMetadataLoader should skip listing
files in hidden and tmp directories
..
Patch Set 2:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@580
PS2, Line 580: /**
:* Filter interface for a FileStatus. Useful for filtering out
disinteresting
:* FileStatuses from a given list
:*/
: public interface FileStatusFilter {
:
: /**
: * Given a FileStatus returns if the filter accepts it or not
: * @param fileStatus
: * @return true if the filter accepts it, else false
: */
: boolean accept(FileStatus fileStatus);
: }
:
: /**
:* FileStatusFilter implementation which is
:* useful to filter out hidden directories and temporary
staging directories
:* which tools like Hive create in the table/partition
directories when a query is
:* inserting data into them.
:*/
: public static final FileStatusFilter HIDDEN_DIRECTORIES_FILTER
= fileStatus -> {
: if (!fileStatus.isDirectory()) return false;
: String filename = fileStatus.getPath().getName();
: return filename.startsWith(".") ||
filename.startsWith("_tmp.");
: };
You could also implement a Predicate overriding test().
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@656
PS2, Line 656: private final boolean isRecursive_;
I think the refactor is confusing. RecursiingIterator can also have
isRecursive_ = false? Can we clean it up a bit?
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@716
PS2, Line 716: { return; }
nit: no braces
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
File fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java:
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@93
PS2, Line 93: hdfs://localhost:20500/
Do we need these qualifiers? I think the Configuration should resolve it to an
appropriate file system
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@100
PS2, Line 100: dstFs.deleteOnExit(tmpTestPath);
Do you need wrap the fs in try-with-resources for this to work?
http://gerrit.cloudera.org:8080/#/c/13665/2/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@112
PS2, Line 112: 24
Instead, we could try loading sourcePath and substitute the values?
--
To view, visit http://gerrit.cloudera.org:8080/13665
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa
Gerrit-Change-Number: 13665
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 19 Jun 2019 01:41:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Fang-Yu Rao has uploaded a new patch set (#21). (
http://gerrit.cloudera.org:8080/12974 )
Change subject: IMPALA-7608: Estimate row count from file size when no stats
available
..
IMPALA-7608: Estimate row count from file size when no stats available
Added the feature that computes an estimated number of rows in the current
hdfs table if the statistics for the cardinality of the current hdfs table is
not
available.
Also added an additional query option to revert the change in case of
regression.
Testing:
(1) In CardinalityTest.java, replaced the original statement
"verifyCardinality("SELECT a FROM functional.tinytable", -1);" in
the method testBasicsWithoutStats() with
"verifyCardinality("SELECT a FROM functional.tinytable", 2);".
(2) In CarginalityTest.java, added more tests to check the cardinality
of most PlanNode implementations. For each tested PlanNode, the behaviors
before and after we disable the feature are both tested.
(3) In set.test, modified three related test cases to make sure that
the added query option is included after executing "set all" in various
scenarios.
(4) There are 8 JUnit tests in PlannerTest.java that would produce different
distributed query plans when this feature is enabled. Added an additional
JUnit test for each of those 8 affected JUnit tests when this feature is
enabled. Specifically, each tested query in a newly added test files involves
at least one hdfs table without available statistics.
(5) There are 5 Python end to end tests that consist of queries that would
produce different results. Added an additional query for each affected query
when this feature is disabled.
Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsCompression.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
A
testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/joins-hdfs-num-rows-est-enabled.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
A
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements-hdfs-num-rows-est-enabled.test
A
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
A
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
25 files changed, 3,042 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/12974/21
--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 21
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8667. Remove --pull incremental stats flag
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13671 ) Change subject: IMPALA-8667. Remove --pull_incremental_stats flag .. IMPALA-8667. Remove --pull_incremental_stats flag This flag was added as a "chicken bit" -- so we could disable the new feature if we had some problems with it. It's been out in the wild for a number of months and we haven't seen any such problems, so at this point let's stop maintaining the old code path. Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 Reviewed-on: http://gerrit.cloudera.org:8080/13671 Reviewed-by: Bharath Vissapragada Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M docs/topics/impala_metadata.xml M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M tests/common/custom_cluster_test_suite.py M tests/conftest.py D tests/custom_cluster/test_incremental_stats.py 15 files changed, 36 insertions(+), 180 deletions(-) Approvals: Bharath Vissapragada: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 Gerrit-Change-Number: 13671 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8667. Remove --pull incremental stats flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13671 ) Change subject: IMPALA-8667. Remove --pull_incremental_stats flag .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 Gerrit-Change-Number: 13671 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 19 Jun 2019 01:06:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4498/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 19 Jun 2019 00:52:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 4: Code-Review+1 (4 comments) lgtm http://gerrit.cloudera.org:8080/#/c/13664/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@243 PS4, Line 243: AtomicInteger piggybackSuccessCountForTests = new AtomicInteger(); nit: final http://gerrit.cloudera.org:8080/#/c/13664/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@250 PS4, Line 250: AtomicInteger piggybackExceptionCountForTests = new AtomicInteger(); nit: final http://gerrit.cloudera.org:8080/#/c/13664/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@520 PS4, Line 520: " nit: Also log piggyBacked for completeness? http://gerrit.cloudera.org:8080/#/c/13664/4/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java: http://gerrit.cloudera.org:8080/#/c/13664/4/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@289 PS4, Line 289: if (testSuccessCase) throw e; Should we also assert that in either case, the future is removed from the cache ? (and replaced by something meaningful in the success case) -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 19 Jun 2019 00:49:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3674/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 00:30:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 5: Code-Review+2 gvo failed on clang-tidy due to unused variable carrying +2 forward -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 19 Jun 2019 00:16:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13583 to look at the new patch set (#5). Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. IMPALA-7467: Port ExecQueryFInstances to krpc This patch ports the ExecQueryFInstances rpc to use KRPC. The parameters for this call contain a huge number of Thrift structs (eg. everything related to TPlanNode and TExpr), so to avoid converting all of these to protobuf and the resulting effect that would have on the FE and catalog, this patch stores most of the parameters in a sidecar (in particular the TQueryCtx, TPlanFragmentCtx's, and TPlanFragmentInstanceCtx's). Testing: - Passed a full exhaustive run on the minicluster. Set up a ten node cluster with tpch 500: - Ran perf tests: 3 iterations per tpch query, 4 concurrent streams, no perf change. - Ran the stress test for 1000 queries, passed. Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 --- M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/test-env.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M common/protobuf/control_service.proto M common/thrift/ImpalaInternalService.thrift M tests/custom_cluster/test_rpc_timeout.py 15 files changed, 245 insertions(+), 188 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13583/5 -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4494/ -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 19 Jun 2019 00:15:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3673/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 19 Jun 2019 00:07:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4497/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 16 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 19 Jun 2019 00:03:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4496/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 23:57:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Hello Lars Volker, Tim Armstrong, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13545 to look at the new patch set (#6). Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. IMPALA-8630: Hash the full path when calculating consistent remote placement Consistent remote placement currently uses the relative filename within a partition for the consistent hash. If the relative filenames for different partitions have a simple naming scheme, then multiple partitions may have files of the same name. This is true for some tables written by Hive (e.g. in our minicluster the tpcds.store_sales has this problem). This can lead to unbalanced placement of remote ranges. This adds a partition_path_hash to the THdfsFileSplit and THdfsFileSplitGeneratorSpec, calculated in the frontend (which has all of the partition information). The scheduler hashes this in addition to the relative path. Testing: - Added several new scheduler tests that verify the consistent remote scheduling see blocks with different relative paths, partition paths, or offsets as distinct. - Ran core tests Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b --- M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java 7 files changed, 277 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13545/6 -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 23:37:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 23:37:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4495/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 10 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 23:37:28 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8669 Support giving a terminal commit in compare branches.py
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13660 ) Change subject: IMPALA-8669 Support giving a terminal commit in compare_branches.py .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I7a4dfae26f8f616c6f550114edfbafd6fc1c068d Gerrit-Change-Number: 13660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 23:31:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13583 to look at the new patch set (#4). Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. IMPALA-7467: Port ExecQueryFInstances to krpc This patch ports the ExecQueryFInstances rpc to use KRPC. The parameters for this call contain a huge number of Thrift structs (eg. everything related to TPlanNode and TExpr), so to avoid converting all of these to protobuf and the resulting effect that would have on the FE and catalog, this patch stores most of the parameters in a sidecar (in particular the TQueryCtx, TPlanFragmentCtx's, and TPlanFragmentInstanceCtx's). Testing: - Passed a full exhaustive run on the minicluster. Set up a ten node cluster with tpch 500: - Ran perf tests: 3 iterations per tpch query, 4 concurrent streams, no perf change. - Ran the stress test for 1000 queries, passed. Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 --- M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/test-env.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M common/protobuf/control_service.proto M common/thrift/ImpalaInternalService.thrift M tests/custom_cluster/test_rpc_timeout.py 15 files changed, 243 insertions(+), 184 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13583/4 -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/13583 )
Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13583/2/be/src/runtime/backend-client.h
File be/src/runtime/backend-client.h:
http://gerrit.cloudera.org:8080/#/c/13583/2/be/src/runtime/backend-client.h@48
PS2, Line 48: /// Callers of TransmitData() should provide their own counter
to measure the data
: /// transmission time.
: void
SetTransmitDataCounter(RuntimeProfile::ConcurrentTimerCounter* csw) {
: DCHECK(transmit_csw_ == NULL);
: transmit_csw_ = csw;
: }
:
: /// ImpalaBackendClient is shared by multiple queries. It's the
caller's responsibility
: /// to reset the counter after data transmission.
: void ResetTransmitDataCounter() {
: transmit_csw_ = NULL;
: }
> Not your change but I believe these are not needed anymore. May as well del
Done
http://gerrit.cloudera.org:8080/#/c/13583/2/common/protobuf/control_service.proto
File common/protobuf/control_service.proto:
http://gerrit.cloudera.org:8080/#/c/13583/2/common/protobuf/control_service.proto@235
PS2, Line 235: 5
> Why the jump from 2 to 5 ?
Done
--
To view, visit http://gerrit.cloudera.org:8080/13583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570
Gerrit-Change-Number: 13583
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Tue, 18 Jun 2019 23:26:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4494/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 18 Jun 2019 23:26:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3672/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 23:23:26 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8669 Support giving a terminal commit in compare branches.py
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13660 ) Change subject: IMPALA-8669 Support giving a terminal commit in compare_branches.py .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/13660/1/bin/compare_branches.py File bin/compare_branches.py: http://gerrit.cloudera.org:8080/#/c/13660/1/bin/compare_branches.py@105 PS1, Line 105: h > flake8: E128 continuation line under-indented for visual indent Ignored this to keep original code style http://gerrit.cloudera.org:8080/#/c/13660/1/bin/compare_branches.py@166 PS1, Line 166: > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/13660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I7a4dfae26f8f616c6f550114edfbafd6fc1c068d Gerrit-Change-Number: 13660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 18 Jun 2019 23:21:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@455 PS2, Line 455: //load to be triggered, which sees the post-invalidated data in catalogd. > Yeah, I mean even a unit test would be helpful. The logic seems simple enou I verified that the new e2e test does cover these cases, but also added a specific targeted unit test along with some test-only counters. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 22:44:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Hello Bharath Vissapragada, Vihang Karajgaonkar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13664 to look at the new patch set (#4). Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. IMPALA-7534. Handle invalidation races in CatalogdMetaProvider This handles a race condition in which a cache invalidation concurrent with a cache load would potentially be skipped, causing out-of-date data to persist in the cache. This would present itself as spurious "table not found" errors. A new test case triggers the issue reliably by injecting latency into the metadata fetch RPC and running DDLs concurrently on the same database across 8 threads. With the fix, the test passes reliably. Another option to fix this might have been to switch to Caffeine instead of Guava's loading cache. However, Caffeine requires Java 8, and LocalCatalog is being backported to Impala 2.x which still can run on Java 7. So, working around the Guava issue will make backporting (and future backports) easier. Change-Id: I70f377db88e204825a909389f28dc3451815235c --- M be/src/exec/catalog-op-executor.cc M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M tests/custom_cluster/test_local_catalog.py 4 files changed, 243 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/13664/4 -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3671/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 22:25:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3670/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 22:16:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3668/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 22:09:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3669/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 22:09:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3667/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 22:03:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3666/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 21:52:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4490/ -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 9 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 18 Jun 2019 21:51:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 3: Code-Review+1 Carry Tim's +1. -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 18 Jun 2019 21:48:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7802: Close connections of idle client sessions
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13607 ) Change subject: IMPALA-7802: Close connections of idle client sessions .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13607/2/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/13607/2/be/src/service/impala-server.cc@219 PS2, Line 219: DEFINE_int32(idle_client_poll_period_ms, 3, "The poll period, in milliseconds, after " Maybe make this seconds not ms? I don't think there's a reason anyone would need to set this lower than 1s, and a very small value could end up causing contention issues with the locks that are taken. -- To view, visit http://gerrit.cloudera.org:8080/13607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38 Gerrit-Change-Number: 13607 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Manish Maheshwari Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 18 Jun 2019 21:44:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12884 ) Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. Patch Set 25: > (1 comment) > > I chickened out on changing the default to HS2 - I think the chance > of breaking workflows is non-zero, even though the change is mostly > transparent so we should hold off on flipping the default until a > breaking version. Let me know if this makes sense to you. Sure, I think that's reasonable. -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 25 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 21:44:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4493/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 21:43:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Abhishek Rawat has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. IMPALA-8617: Add support for lz4 in parquet A new enum value LZ4_BLOCKED was added to the THdfsCompression enum, to distinguish it from the existing LZ4 codec. LZ4_BLOCKED codec represents the block compression scheme used by Hadoop. Its similar to SNAPPY_BLOCKED as far as the block format is concerned, with the only difference being the codec used for compression and decompression. Added Lz4BlockCompressor and Lz4BlockDecompressor classes for compressing and decompressing parquet data using Hadoop's lz4 block compression scheme. The Lz4BlockCompressor treats the input as a single block and generates a compressed block with following layout <4 byte big endian uncompressed size> <4 byte big endian compressed size> The hdfs parquet table writer should call the Lz4BlockCompressor using the ideal input size (unit of compression in parquet is a page), and so the Lz4BlockCompressor does not further break down the input into smaller blocks. The Lz4BlockDecompressor on the other hand should be compatible with blocks written by Impala and other engines in Hadoop ecosystem. It can decompress compressed data in following format <4 byte big endian uncompressed size> <4 byte big endian compressed size> ... <4 byte big endian compressed size> ... Externally users can now set the lz4 codec for parquet using: set COMPRESSION_CODEC=lz4 This gets translated into LZ4_BLOCKED codec for the HdfsParquetTableWriter. Similarly, when reading lz4 compressed parquet data, the LZ4_BLOCKED codec is used. Testing: - Added unit tests for LZ4_BLOCKED in decompress-test.cc - Added unit tests for Hadoop compatibility in decompress-test.cc, basically being able to decompress an outer block with multiple inner blocks (the Lz4BlockDecompressor description above) - Added interoperability tests for Hive and Impala for all parquet codecs. New test added to tests/custom_cluster/test_hive_parquet_codec_interop.py Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/service/query-options-test.cc M be/src/util/codec.cc M be/src/util/compress.cc M be/src/util/compress.h M be/src/util/decompress-test.cc M be/src/util/decompress.cc M be/src/util/decompress.h M common/thrift/CatalogObjects.thrift M common/thrift/generate_error_codes.py M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/common/test_dimensions.py A tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/query_test/test_insert.py M tests/query_test/test_insert_parquet.py 18 files changed, 440 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/13582/9 -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 9 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Hello Tim Armstrong, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13545 to look at the new patch set (#4). Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. IMPALA-8630: Hash the full path when calculating consistent remote placement Consistent remote placement currently uses the relative filename within a partition for the consistent hash. If the relative filenames for different partitions have a simple naming scheme, then multiple partitions may have files of the same name. This is true for some tables written by Hive (e.g. in our minicluster the tpcds.store_sales has this problem). This can lead to unbalanced placement of remote ranges. This adds a partition_path_hash to the THdfsFileSplit and THdfsFileSplitGeneratorSpec, calculated in the frontend (which has all of the partition information). The scheduler hashes this in addition to the relative path. Testing: - Added several new scheduler tests that verify the consistent remote scheduling see blocks with different relative paths, partition paths, or offsets as distinct. - Ran core tests Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b --- M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java 6 files changed, 276 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13545/4 -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12884 ) Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. Patch Set 25: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 25 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 21:37:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 10: Yongzhi, feel free to rebase when you get a chance. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 18 Jun 2019 21:33:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4492/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 21:28:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 22 files changed, 223 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/11 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 8: (2 comments) http://gerrit.cloudera.org:8080/#/c/13582/8/tests/custom_cluster/test_hive_parquet_codec_interop.py File tests/custom_cluster/test_hive_parquet_codec_interop.py: http://gerrit.cloudera.org:8080/#/c/13582/8/tests/custom_cluster/test_hive_parquet_codec_interop.py@26 PS8, Line 26: from tests.common.test_vector import ImpalaTestDimension flake8: F401 'tests.common.test_vector.ImpalaTestDimension' imported but unused http://gerrit.cloudera.org:8080/#/c/13582/8/tests/custom_cluster/test_hive_parquet_codec_interop.py@66 PS8, Line 66: a flake8: E501 line too long (101 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 21:28:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 16: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4487/ -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 16 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 21:27:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Abhishek Rawat has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. IMPALA-8617: Add support for lz4 in parquet A new enum value LZ4_BLOCKED was added to the THdfsCompression enum, to distinguish it from the existing LZ4 codec. LZ4_BLOCKED codec represents the block compression scheme used by Hadoop. Its similar to SNAPPY_BLOCKED as far as the block format is concerned, with the only difference being the codec used for compression and decompression. Added Lz4BlockCompressor and Lz4BlockDecompressor classes for compressing and decompressing parquet data using Hadoop's lz4 block compression scheme. The Lz4BlockCompressor treats the input as a single block and generates a compressed block with following layout <4 byte big endian uncompressed size> <4 byte big endian compressed size> The hdfs parquet table writer should call the Lz4BlockCompressor using the ideal input size (unit of compression in parquet is a page), and so the Lz4BlockCompressor does not further break down the input into smaller blocks. The Lz4BlockDecompressor on the other hand should be compatible with blocks written by Impala and other engines in Hadoop ecosystem. It can decompress compressed data in following format <4 byte big endian uncompressed size> <4 byte big endian compressed size> ... <4 byte big endian compressed size> ... Externally users can now set the lz4 codec for parquet using: set COMPRESSION_CODEC=lz4 This gets translated into LZ4_BLOCKED codec for the HdfsParquetTableWriter. Similarly, when reading lz4 compressed parquet data, the LZ4_BLOCKED codec is used. Testing: - Added unit tests for LZ4_BLOCKED in decompress-test.cc - Added unit tests for Hadoop compatibility in decompress-test.cc, basically being able to decompress an outer block with multiple inner blocks (the Lz4BlockDecompressor description above) - Added interoperability tests for Hive and Impala for all parquet codecs. New test added to tests/custom_cluster/test_hive_parquet_codec_interop.py Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/service/query-options-test.cc M be/src/util/codec.cc M be/src/util/compress.cc M be/src/util/compress.h M be/src/util/decompress-test.cc M be/src/util/decompress.cc M be/src/util/decompress.h M common/thrift/CatalogObjects.thrift M common/thrift/generate_error_codes.py M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/common/test_dimensions.py A tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/query_test/test_insert.py M tests/query_test/test_insert_parquet.py 18 files changed, 440 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/13582/8 -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 8 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/13582 ) Change subject: IMPALA-8617: Add support for lz4 in parquet .. Patch Set 7: (2 comments) Updated the tests/custom_cluster/test_hive_parquet_codec_interop.py to loop through the codecs inside the test case and avoid multiple impalad restarts. http://gerrit.cloudera.org:8080/#/c/13582/7/tests/custom_cluster/test_hive_parquet_codec_interop.py File tests/custom_cluster/test_hive_parquet_codec_interop.py: http://gerrit.cloudera.org:8080/#/c/13582/7/tests/custom_cluster/test_hive_parquet_codec_interop.py@32 PS7, Line 32: class TestParquetInterop(CustomClusterTestSuite): > How much time do these tests add to the runtime? My main concern is just th These tests take `2m14.625s` on my dev box. I moved the iteration over compression_codecs inside the test case and it reduces the time by ~1 minute: `1m21.112s`. http://gerrit.cloudera.org:8080/#/c/13582/7/tests/custom_cluster/test_hive_parquet_codec_interop.py@45 PS7, Line 45: cls.ImpalaTestMatrix.add_dimension( > We could reduce the number of parameter combinations by iterating over the Done -- To view, visit http://gerrit.cloudera.org:8080/13582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c Gerrit-Change-Number: 13582 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 21:26:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/13386/8/tests/query_test/test_kudu.py File tests/query_test/test_kudu.py: http://gerrit.cloudera.org:8080/#/c/13386/8/tests/query_test/test_kudu.py@123 PS8, Line 123: @pytest.mark.skipif(IMPALA_TEST_CLUSTER_PROPERTIES.is_remote_cluster(), Added this back in when I rebased. It was removed in https://gerrit.cloudera.org/#/c/13409/1..2/tests/query_test/test_kudu.py but I can't see how the test would work on a remote cluster. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 21:23:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Hello David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13386 to look at the new patch set (#8). Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. IMPALA-8553,IMPALA-8552: fix checks for remote cluster Apparently IMPALA_REMOTE_URL is not generally used for remote cluster tests: only --testing_remote_cluster is reliably set. Fix the is_remote_cluster() implementation to take into account REMOTE_DATA_LOAD and --testing_remote_cluster in addition to IMPALA_REMOTE_URL. Consistently use is_remote_cluster() in other tests instead of checking the pytest flag directly. There were a few lifecycle headaches with how ImpalaTestClusterProperties is used: * common.environ is imported from conftest, which means that the top-level code in the file runs *before* pytest command-line arguments have been registered and parsed. * ImpalaTestClusterProperties is used by various code, like build_flavor_timeout(), which runs before pytest command-line arguments have been parsed. * ImpalaTestClusterProperties is called from non-pytest scripts like start-impala-cluster.py, so the command-line arguments are not available. I dealt with the above challenges by making a few changes to do the detection later: * Lazily initializing a singleton ImpalaTestClusterProperties. This was not strictly necessary but makes the whole problem less sensitive to import order and module dependencies. * Adding cluster_properties fixture to make ImpalaTestClusterProperties available in tests without additional boilerplate. * Removing the caching of the local/remote build calculation. ImpalaTestClusterProperties is instantiated outside of python tests, but is_remote_cluster() is only called from python tests, so if we check flags in is_remote_cluster() we'll get the right results reliably. As a workaround to unblock remote tests, also assume catalog_v1 if accessing the web UI fails. Testing: Ran core tests against a regular minicluster. Ran tests against a remote cluster Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 --- M tests/common/environ.py M tests/common/skip.py M tests/conftest.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_automatic_invalidation.py M tests/hs2/test_hs2.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_kudu.py M tests/query_test/test_mt_dop.py M tests/query_test/test_spilling.py M tests/shell/test_shell_commandline.py M tests/shell/util.py M tests/webserver/test_web_pages.py 16 files changed, 131 insertions(+), 91 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13386/8 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. Patch Set 4: (1 comment) For web UI changes, I generally find it useful to attach screenshots to the JIRA of what exactly changed. It helps document things more clearly. http://gerrit.cloudera.org:8080/#/c/13670/4/be/src/util/default-path-handlers.cc File be/src/util/default-path-handlers.cc: http://gerrit.cloudera.org:8080/#/c/13670/4/be/src/util/default-path-handlers.cc@a134 PS4, Line 134: A nice thing about this DCHECK is that it prevented callers from inadvertently passing in a nullptr for mem_tracker, but now that we have removed the DCHECK, a client might accidentally pass in a nullptr when they didn't intend to. I would suggest re-factoring the code so there are multiple MemUsageHandler methods, one that takes in a mem_tracker, and one that does not. -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 18 Jun 2019 21:15:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 7: Implemented your suggestion in ps7 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 18 Jun 2019 21:12:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Hello David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13386 to look at the new patch set (#7). Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. IMPALA-8553,IMPALA-8552: fix checks for remote cluster Apparently IMPALA_REMOTE_URL is not generally used for remote cluster tests: only --testing_remote_cluster is reliably set. Fix the is_remote_cluster() implementation to take into account REMOTE_DATA_LOAD and --testing_remote_cluster in addition to IMPALA_REMOTE_URL. Consistently use is_remote_cluster() in other tests instead of checking the pytest flag directly. There were a few lifecycle headaches with how ImpalaTestClusterProperties is used: * common.environ is imported from conftest, which means that the top-level code in the file runs *before* pytest command-line arguments have been registered and parsed. * ImpalaTestClusterProperties is used by various code, like build_flavor_timeout(), which runs before pytest command-line arguments have been parsed. * ImpalaTestClusterProperties is called from non-pytest scripts like start-impala-cluster.py, so the command-line arguments are not available. I dealt with the above challenges by making a few changes to do the detection later: * Lazily initializing a singleton ImpalaTestClusterProperties. This was not strictly necessary but makes the whole problem less sensitive to import order and module dependencies. * Adding cluster_properties fixture to make ImpalaTestClusterProperties available in tests without additional boilerplate. * Removing the caching of the local/remote build calculation. ImpalaTestClusterProperties is instantiated outside of python tests, but is_remote_cluster() is only called from python tests, so if we check flags in is_remote_cluster() we'll get the right results reliably. As a workaround to unblock remote tests, also assume catalog_v1 if accessing the web UI fails. Testing: Ran core tests against a regular minicluster. Ran tests against a remote cluster Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 --- M tests/common/environ.py M tests/common/skip.py M tests/conftest.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_automatic_invalidation.py M tests/hs2/test_hs2.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_kudu.py M tests/query_test/test_mt_dop.py M tests/query_test/test_spilling.py M tests/shell/test_shell_commandline.py M tests/shell/util.py M tests/webserver/test_web_pages.py 16 files changed, 129 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13386/7 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13673 )
Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant
..
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG@7
PS3, Line 7: [IMPALA-8587]
nit: use IMPALA-8587 format
http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG@20
PS3, Line 20: They would see no results. After this change, the user will see
database
: level privileges when executing the previous statement. If a user
has
: SELECT privilege on DATABASE and on TABLE and issues a show grant
on
: TABLE, they will only see the SELECT privilege for TABLE. Users
will not
: see multiple instances of SELECT or any other privilege type in a
SHOW
: GRANT statemenet.
Show what the new output looks like. It'll be much easier to understand.
We also need more explanation about this especially since this is relates to
how the Ranger policy engine works.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@330
PS3, Line 330: !=
Use equals(). != or == is for reference equality. Sometimes you get lucky
because string intern, but we shouldn't rely on that.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@329
PS3, Line 329: for (RangerResultRow row : resultRows) {
: if (row.column_ != "*" && !row.column_.isEmpty()) {
: resourceResult.addColumnResult(row);
: } else if (row.table_ != "*" && !row.table_.isEmpty()) {
: resourceResult.addTableResult(row);
: } else if (row.database_ != "*" &&
!row.database_.isEmpty()) {
: resourceResult.addDatabaseResult(row);
: } else {
: resourceResult.addServerResult(row);
: }
: }
Can you add a comment for this logic? It's not quite clear to me what it's
trying to do.
http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@422
PS3, Line 422:
nit: overly indented, use 4 spaces
--
To view, visit http://gerrit.cloudera.org:8080/13673
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c
Gerrit-Change-Number: 13673
Gerrit-PatchSet: 3
Gerrit-Owner: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 18 Jun 2019 21:08:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@442 PS2, Line 442: Future existing = (Future)inCache; > yea, I was wondering whether to add another separate metric for "piggybacke Yeah I'm also not sure how useful it would be, I think for most purposes this is a cache miss. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 21:01:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3665/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 20:58:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 10: Code-Review+1 Looks like this needs to be rebased, but overall LGTM. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 18 Jun 2019 20:55:39 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3664/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 20:48:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@455 PS2, Line 455: // If there was an exception, remove it from the map so that any later loads > I'm guessing this gets covered by our stress test which does concurrent que Yeah, I mean even a unit test would be helpful. The logic seems simple enough to verify by inspection, but I can imagine it might be easy to break the error propagation with other changes if we're not careful. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 20:45:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@442 PS2, Line 442: // > We're not setting hit = true, so we're counting this case as a cache miss? yea, I was wondering whether to add another separate metric for "piggybacked hits" or something, but wasn't sure it was worth it. What do you think? http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@455 PS2, Line 455: boolean hit = false; > Do we have tests for these exceptional code paths? Including the case when I'm guessing this gets covered by our stress test which does concurrent queries and invalidates. I'll double check that this is actually covered and add another stress test if not. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 20:43:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3663/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 20:42:45 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3662/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 1 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 20:42:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Hello Bharath Vissapragada, Vihang Karajgaonkar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13664 to look at the new patch set (#3). Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. IMPALA-7534. Handle invalidation races in CatalogdMetaProvider This handles a race condition in which a cache invalidation concurrent with a cache load would potentially be skipped, causing out-of-date data to persist in the cache. This would present itself as spurious "table not found" errors. A new test case triggers the issue reliably by injecting latency into the metadata fetch RPC and running DDLs concurrently on the same database across 8 threads. With the fix, the test passes reliably. Another option to fix this might have been to switch to Caffeine instead of Guava's loading cache. However, Caffeine requires Java 8, and LocalCatalog is being backported to Impala 2.x which still can run on Java 7. So, working around the Guava issue will make backporting (and future backports) easier. Change-Id: I70f377db88e204825a909389f28dc3451815235c --- M be/src/exec/catalog-op-executor.cc M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/custom_cluster/test_local_catalog.py 3 files changed, 159 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/13664/3 -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@442 PS2, Line 442: // We're not setting hit = true, so we're counting this case as a cache miss? I guess that makes sense in some way, since we'll be incurring some of the loading delay. Might be clearer if this was explained, maybe in a comment attached to "hit". Or maybe there's a better name for hit, like immediateHit. http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@455 PS2, Line 455: boolean hit = false; Do we have tests for these exceptional code paths? Including the case when multiple threads end up getting the same exception. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 20:40:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/13583 )
Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc
..
Patch Set 3: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13583/2/be/src/runtime/backend-client.h
File be/src/runtime/backend-client.h:
http://gerrit.cloudera.org:8080/#/c/13583/2/be/src/runtime/backend-client.h@48
PS2, Line 48: /// Callers of TransmitData() should provide their own counter
to measure the data
: /// transmission time.
: void
SetTransmitDataCounter(RuntimeProfile::ConcurrentTimerCounter* csw) {
: DCHECK(transmit_csw_ == NULL);
: transmit_csw_ = csw;
: }
:
: /// ImpalaBackendClient is shared by multiple queries. It's the
caller's responsibility
: /// to reset the counter after data transmission.
: void ResetTransmitDataCounter() {
: transmit_csw_ = NULL;
: }
Not your change but I believe these are not needed anymore. May as well delete
them while you are here.
http://gerrit.cloudera.org:8080/#/c/13583/2/common/protobuf/control_service.proto
File common/protobuf/control_service.proto:
http://gerrit.cloudera.org:8080/#/c/13583/2/common/protobuf/control_service.proto@235
PS2, Line 235: 5
Why the jump from 2 to 5 ?
--
To view, visit http://gerrit.cloudera.org:8080/13583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570
Gerrit-Change-Number: 13583
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Comment-Date: Tue, 18 Jun 2019 20:37:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3661/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 18 Jun 2019 20:32:09 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13673/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/13673/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@341 PS2, Line 341: resourceResult.getResultRows().forEach(principal -> resultSet.add(principal.toResultRow())); line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/13673/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13673/2/tests/authorization/test_ranger.py@195 PS2, Line 195: t flake8: E501 line too long (96 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13673/2/tests/authorization/test_ranger.py@350 PS2, Line 350: ) flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 20:01:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Austin Nobis has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. [IMPALA-8587] Show inherited privileges with Ranger show grant Previously when executing a show grant statement on a resource with Ranger authorization enabled, Impala would not show inherited privileges. For example, if a user had database level privileges such as: GRANT SELECT ON DATABASE db TO USER user; If a user then requested table level privileges such as: SHOW GRANT USER user ON TABLE db.table; They would see no results. After this change, the user will see database level privileges when executing the previous statement. If a user has SELECT privilege on DATABASE and on TABLE and issues a show grant on TABLE, they will only see the SELECT privilege for TABLE. Users will not see multiple instances of SELECT or any other privilege type in a SHOW GRANT statemenet. Testing - Ran all FE tests - Ran all authorization E2E tests - Added E2E tests in test_ranger verifying functionality Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 144 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/13673/2 -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/13673/1/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/13673/1/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@341 PS1, Line 341: resourceResult.getResultRows().forEach(principal -> resultSet.add(principal.toResultRow())); line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/13673/1/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13673/1/tests/authorization/test_ranger.py@195 PS1, Line 195: t flake8: E501 line too long (96 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13673/1/tests/authorization/test_ranger.py@350 PS1, Line 350: ) flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 1 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 20:01:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13673 to look at the new patch set (#3). Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. [IMPALA-8587] Show inherited privileges with Ranger show grant Previously when executing a show grant statement on a resource with Ranger authorization enabled, Impala would not show inherited privileges. For example, if a user had database level privileges such as: GRANT SELECT ON DATABASE db TO USER user; If a user then requested table level privileges such as: SHOW GRANT USER user ON TABLE db.table; They would see no results. After this change, the user will see database level privileges when executing the previous statement. If a user has SELECT privilege on DATABASE and on TABLE and issues a show grant on TABLE, they will only see the SELECT privilege for TABLE. Users will not see multiple instances of SELECT or any other privilege type in a SHOW GRANT statemenet. Testing - Ran all FE tests - Ran all authorization E2E tests - Added E2E tests in test_ranger verifying functionality Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 147 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/13673/3 -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Austin Nobis has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13673 Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. [IMPALA-8587] Show inherited privileges with Ranger show grant Previously when executing a show grant statement on a resource with Ranger authorization enabled, Impala would not show inherited privileges. For example, if a user had database level privileges such as: GRANT SELECT ON DATABASE db TO USER user; If a user then requested table level privileges such as: SHOW GRANT USER user ON TABLE db.table; They would see no results. After this change, the user will see database level privileges when executing the previous statement. If a user has SELECT privilege on DATABASE and on TABLE and issues a show grant on TABLE, they will only see the SELECT privilege for TABLE. Users will not see multiple instances of SELECT or any other privilege type in a SHOW GRANT statemenet. Testing - Ran all FE tests - Ran all authorization E2E tests - Added E2E tests in test_ranger verifying functionality Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 147 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/13673/1 -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 1 Gerrit-Owner: Austin Nobis
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG@7 PS3, Line 7: > remove ... Done http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc File be/src/util/default-path-handlers.cc: http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc@135 PS3, Line 135: if (mem_tracker != nullptr > It's less nested to do if (mem_tracker == nullptr) return; The DCHECK was removed because it was only guarding against null MemTracker. The decision whether to print the MemTracker metrics or not was added instead. I might be misunderstanding something. -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 18 Jun 2019 19:50:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13670 to look at the new patch set (#4). Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. IMPALA-7734: Catalog and Statestore memz page shows useless memory The 'MemTracker' is removed from StatestoreD main and CatalogD main as it is not used. When MemTracker is not available printing those memory metrics are skipped. Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 --- M be/src/catalog/catalogd-main.cc M be/src/statestore/statestored-main.cc M be/src/util/default-path-handlers.cc M www/memz.tmpl 4 files changed, 26 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/13670/4 -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@687 PS2, Line 687: RuntimeException > Do you know if we ever interrupt threads? Otherwise I don't think this intr Yea, don't think we interrupt. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 19:46:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8667. Remove --pull incremental stats flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13671 ) Change subject: IMPALA-8667. Remove --pull_incremental_stats flag .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4491/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 Gerrit-Change-Number: 13671 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 19:44:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8667. Remove --pull incremental stats flag
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13671 ) Change subject: IMPALA-8667. Remove --pull_incremental_stats flag .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 Gerrit-Change-Number: 13671 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 19:40:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3660/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 18 Jun 2019 19:29:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@427 PS2, Line 427: private ValueType loadWithCaching(String itemString, > I think this could use a method comment since the logic is non-trivial (pro Ack http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@435 PS2, Line 435: Object inCache = cache_.get(key, () -> f); > Just for my understanding, here we are relying on the inherent thread-safet yea, will add a comment http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@573 PS2, Line 573: Uninterruptibles.sleepUninterruptibly(10, TimeUnit.MILLISECONDS); > Remove? woops! http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@687 PS2, Line 687: RuntimeException > Does this mean the query is not eventually retried? Do you know if we ever interrupt threads? Otherwise I don't think this introduces any actual possibility for an exception. -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 19:21:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8584: Add cookie support to the HTTP HS2 server
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13672 ) Change subject: IMPALA-8584: Add cookie support to the HTTP HS2 server .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3659/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I647c06f94ef91aa3b6413e91576c4ec506ed57f4 Gerrit-Change-Number: 13672 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 19:20:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8584: Add cookie support to the HTTP HS2 server
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13672 )
Change subject: IMPALA-8584: Add cookie support to the HTTP HS2 server
..
Patch Set 2:
(5 comments)
didn't look in detail yet, a couple high-level questions first
http://gerrit.cloudera.org:8080/#/c/13672/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13672/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-8584: Add cookie support to the HTTP HS2 server
Some questions about the design here:
- by persisting the cookies in a bimap, what happens when you have multiple
clients separately authenticating as the same user? It seems like you'd end up
handing the same UUID cookie to all of the remote users, and then they'd all
expire at the same time, causing them to all have to go back to LDAP to
reauthenticate simultaneously, which might be problematic for a large number of
clients. It might be better to just use unique session IDs, rather than
maintaining the reverse mapping from username->uuid.
- Similar question: if Impala is behind a load balancer of some kind, when a
user reconnects, they'll use the same cookie to talk to a different impalad.
That other impalad won't find the cookie and will force them to re-auth each
time they switch backends I guess? In typical deployments, do people reconnect
frequently through the load balancer or tend to use sticky sessions? An
alternative here would be to use HMAC signatures in the cookie, using a secret
shared via the statestore or somesuch.
Any idea what Hive does, for comparison? Would be good to document a couple of
these design points in the commit message or the JIRA.
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/rpc/authentication.cc
File be/src/rpc/authentication.cc:
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/rpc/authentication.cc@546
PS2, Line 546: random_device
Any idea what the source of this entropy is? Is it subject to entropy deletion
DOS attacks? (or entropy depletion associated perf problems?)
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/rpc/authentication.cc@562
PS2, Line 562: cookie = Substitute("impala.hs2.auth=$0", PrintId(cookie_id));
Do we want any expiration on the cookie? Or is the default (session cookie)
appropriate?
Also, I would guess we want this to be an http-only and secure cookie (assuming
https is enabled)? Might be some other settings that would be a good idea for
security: see https://odino.org/security-hardening-http-cookies/ which has a
bunch of recommendations.
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/rpc/cookie-mgr.cc
File be/src/rpc/cookie-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/rpc/cookie-mgr.cc@61
PS2, Line 61: username_to_cookie_.emplace(std::piecewise_construct,
std::forward_as_tuple(username),
hrm, this is odd. YOu can't just do .emplace(username, Cookie(cookie,
UnixMillis()); here?
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/transport/THttpServer.cpp
File be/src/transport/THttpServer.cpp:
http://gerrit.cloudera.org:8080/#/c/13672/2/be/src/transport/THttpServer.cpp@104
PS2, Line 104: cookieValue_ = string(value);
the Cookie header can pass multiple cookies, separated by ';'. It doesn't seem
like that's getting handled here? It might also be possible to pass multiple
Cookie headers, but not sure about that one.
--
To view, visit http://gerrit.cloudera.org:8080/13672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I647c06f94ef91aa3b6413e91576c4ec506ed57f4
Gerrit-Change-Number: 13672
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 18 Jun 2019 19:19:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12974 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 20: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3658/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12974 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a Gerrit-Change-Number: 12974 Gerrit-PatchSet: 20 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 19:15:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Fang-Yu Rao has abandoned this change. ( http://gerrit.cloudera.org:8080/13419 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Abandoned The actual review is: https://gerrit.cloudera.org/#/c/12974/ -- To view, visit http://gerrit.cloudera.org:8080/13419 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ie1b28e56a8a98eaf1871766ad6ca1f62c9688fa7 Gerrit-Change-Number: 13419 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-7534. Handle invalidation races in CatalogdMetaProvider
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13664 ) Change subject: IMPALA-7534. Handle invalidation races in CatalogdMetaProvider .. Patch Set 2: (4 comments) Sorry, I got confused by the comments on the jira. It appeared that they are two different issues but upon looking closely, you seem to be fixing that race. Patch makes sense to me. I have some minor comments. http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@427 PS2, Line 427: private ValueType loadWithCaching(String itemString, I think this could use a method comment since the logic is non-trivial (probably move the cache_ comment to this method)? http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@435 PS2, Line 435: Object inCache = cache_.get(key, () -> f); Just for my understanding, here we are relying on the inherent thread-safety of get(key, callable) right? Even if there are concurrent loadWithCaching() calls, only the first future is added to the cache and all other threads block on its get(). http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@573 PS2, Line 573: Uninterruptibles.sleepUninterruptibly(10, TimeUnit.MILLISECONDS); Remove? http://gerrit.cloudera.org:8080/#/c/13664/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@687 PS2, Line 687: RuntimeException Does this mean the query is not eventually retried? -- To view, visit http://gerrit.cloudera.org:8080/13664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70f377db88e204825a909389f28dc3451815235c Gerrit-Change-Number: 13664 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 18 Jun 2019 18:58:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13419 ) Change subject: IMPALA-7608: Estimate row count from file size when no stats available .. Patch Set 1: This can be abandoned, right? The actual review is: https://gerrit.cloudera.org/#/c/12974/ -- To view, visit http://gerrit.cloudera.org:8080/13419 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie1b28e56a8a98eaf1871766ad6ca1f62c9688fa7 Gerrit-Change-Number: 13419 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 18 Jun 2019 18:56:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory...
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13670 )
Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless
memory...
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG@7
PS3, Line 7: ...
remove ...
http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc@135
PS3, Line 135: if (mem_tracker != NULL) {
It's less nested to do if (mem_tracker == nullptr) return;
--
To view, visit http://gerrit.cloudera.org:8080/13670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23
Gerrit-Change-Number: 13670
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 18 Jun 2019 18:56:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13583 ) Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13583/2/be/src/service/control-service.cc File be/src/service/control-service.cc: http://gerrit.cloudera.org:8080/#/c/13583/2/be/src/service/control-service.cc@146 PS2, Line 146: Status resp_status = > line too long (94 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 18 Jun 2019 18:48:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7467: Port ExecQueryFInstances to krpc
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13583 to look at the new patch set (#3). Change subject: IMPALA-7467: Port ExecQueryFInstances to krpc .. IMPALA-7467: Port ExecQueryFInstances to krpc This patch ports the ExecQueryFInstances rpc to use KRPC. The parameters for this call contain a huge number of Thrift structs (eg. everything related to TPlanNode and TExpr), so to avoid converting all of these to protobuf and the resulting effect that would have on the FE and catalog, this patch stores most of the parameters in a sidecar (in particular the TQueryCtx, TPlanFragmentCtx's, and TPlanFragmentInstanceCtx's). Testing: - Passed a full exhaustive run on the minicluster. Set up a ten node cluster with tpch 500: - Ran perf tests: 3 iterations per tpch query, 4 concurrent streams, no perf change. - Ran the stress test for 1000 queries, passed. Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 --- M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/test-env.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M common/protobuf/control_service.proto M common/thrift/ImpalaInternalService.thrift M tests/custom_cluster/test_rpc_timeout.py 15 files changed, 243 insertions(+), 171 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13583/3 -- To view, visit http://gerrit.cloudera.org:8080/13583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id3f1c6099109bd8e5361188005a7d0e892147570 Gerrit-Change-Number: 13583 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8584: Add cookie support to the HTTP HS2 server
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13672 to look at the new patch set (#2). Change subject: IMPALA-8584: Add cookie support to the HTTP HS2 server .. IMPALA-8584: Add cookie support to the HTTP HS2 server This patch modifies the HTTP HS2 server to accept cookies for authentication in order to avoid having to authenticate every request through LDAP. It adds a flag, --max_cookie_lifetime_s, that determines how long generated cookies are valid for. Setting the flag to 0 disables cookie support. It also adds the metrics: impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure Testing: - Added tests to the FE LDAP tests. Change-Id: I647c06f94ef91aa3b6413e91576c4ec506ed57f4 --- M be/src/rpc/CMakeLists.txt M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/auth-provider.h M be/src/rpc/authentication.cc A be/src/rpc/cookie-mgr.cc A be/src/rpc/cookie-mgr.h M be/src/rpc/thrift-server.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M common/thrift/metrics.json M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java M fe/src/test/java/org/apache/impala/service/JdbcTestBase.java 15 files changed, 486 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13672/2 -- To view, visit http://gerrit.cloudera.org:8080/13672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I647c06f94ef91aa3b6413e91576c4ec506ed57f4 Gerrit-Change-Number: 13672 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8584: Add cookie support to the HTTP HS2 server
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13672 Change subject: IMPALA-8584: Add cookie support to the HTTP HS2 server .. IMPALA-8584: Add cookie support to the HTTP HS2 server This patch modifies the HTTP HS2 server to accept cookies for authentication in order to avoid having to authenticate every request through LDAP. Change-Id: I647c06f94ef91aa3b6413e91576c4ec506ed57f4 --- M be/src/rpc/CMakeLists.txt M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/auth-provider.h M be/src/rpc/authentication.cc A be/src/rpc/cookie-mgr.cc A be/src/rpc/cookie-mgr.h M be/src/rpc/thrift-server.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M common/thrift/metrics.json M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java M fe/src/test/java/org/apache/impala/service/JdbcTestBase.java 15 files changed, 486 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13672/1 -- To view, visit http://gerrit.cloudera.org:8080/13672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I647c06f94ef91aa3b6413e91576c4ec506ed57f4 Gerrit-Change-Number: 13672 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-8667. Remove --pull incremental stats flag
Hello Bharath Vissapragada, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13671 to review the following change. Change subject: IMPALA-8667. Remove --pull_incremental_stats flag .. IMPALA-8667. Remove --pull_incremental_stats flag This flag was added as a "chicken bit" -- so we could disable the new feature if we had some problems with it. It's been out in the wild for a number of months and we haven't seen any such problems, so at this point let's stop maintaining the old code path. Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M docs/topics/impala_metadata.xml M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M tests/common/custom_cluster_test_suite.py M tests/conftest.py D tests/custom_cluster/test_incremental_stats.py 15 files changed, 36 insertions(+), 180 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/13671/1 -- To view, visit http://gerrit.cloudera.org:8080/13671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8878fcd8a2462963c7db3183a003bb9816dda8f9 Gerrit-Change-Number: 13671 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada
