[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 6: No problem, I want to do this the right way. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 05:44:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13507 ) Change subject: IMPALA-8450: Add support for zstd in parquet .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7 Gerrit-Change-Number: 13507 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 05 Jun 2019 05:43:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13507 ) Change subject: IMPALA-8450: Add support for zstd in parquet .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4413/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7 Gerrit-Change-Number: 13507 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 05 Jun 2019 05:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13507 ) Change subject: IMPALA-8450: Add support for zstd in parquet .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3511/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7 Gerrit-Change-Number: 13507 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 05 Jun 2019 05:34:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 05:20:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Abhishek Rawat has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/13507 )
Change subject: IMPALA-8450: Add support for zstd in parquet
..
IMPALA-8450: Add support for zstd in parquet
Makefile was updated to include zstd in the ${IMPALA_HOME}/toolchain
directory. Other changes were made to make zstd headers and libs
accessible.
Class ZstandardCompressor/ZstandardDecompressor was added to provide
interfaces for calling ZSTD_compress/ZSTD_decompress functions. Zstd
supports different compression levels (clevel) from 1 to
ZSTD_maxCLevel(). Zstd also supports -ive clevels, but since the -ive
values represents uncompressed data they won't be supported. The default
clevel is ZSTD_CLEVEL_DEFAULT.
HdfsParquetTableWriter was updated to support ZSTD codec. The
new codecs can be set using existing query option as follows:
set COMPRESSION_CODEC=ZSTD:;
set COMPRESSION_CODEC=ZSTD; // uses ZSTD_CLEVEL_DEFAULT
Testing:
- Added unit test in DecompressorTest class with ZSTD_CLEVEL_DEFAULT
clevel and a random clevel. The test unit decompresses an input
compressed data and validates the result. It also tests for
expected behavior when passing an over/under sized buffer for
decompressing.
- Added unit tests for valid/invalid values for COMPRESSION_CODEC.
- Added e2e test in test_insert_parquet.py which tests writing/read-
ing (null/non-null) data into/from a table (w different data type
columns) using multiple codecs. Other existing e2e tests were
updated to also use parquet/zstd table format.
- Manual interoperability tests were run between Impala and Hive.
Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/experiments/compression-test.cc
M be/src/service/child-query.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/compress.cc
M be/src/util/compress.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M be/src/util/decompress.h
M be/src/util/runtime-profile.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
A cmake_modules/FindZstd.cmake
M common/thrift/CatalogObjects.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_error_codes.py
A
testdata/workloads/functional-query/queries/QueryTest/insert_parquet_multi_codecs.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/common/test_dimensions.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
30 files changed, 497 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13507/6
--
To view, visit http://gerrit.cloudera.org:8080/13507
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
Gerrit-Change-Number: 13507
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13501 ) Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. IMPALA-8578: part 2: move metrics code to .cc files This moves a lot of metric function definitions into .cc files, to reduce the size of compilation units and to reduce the frequency of recompilation required when making changes to metrics. This moves most of the large, non-perf-critical metric functions into .cc files. For template classes, this requires explicitly instantiating all combinations of template parameters that are used in impala, including in tests. Disable weak-template-vtables warning because of spurious warnings on template instantiations. See https://bugs.llvm.org/show_bug.cgi?id=18733 Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Reviewed-on: http://gerrit.cloudera.org:8080/13501 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M .clang-tidy M be/src/exprs/expr-test.cc M be/src/rpc/rpc-trace.cc M be/src/rpc/rpc-trace.h M be/src/rpc/thrift-server.cc M be/src/runtime/bufferpool/reservation-tracker-test.cc M be/src/runtime/coordinator.cc M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/runtime-filter-bank.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/CMakeLists.txt M be/src/util/auth-util.cc A be/src/util/collection-metrics.cc M be/src/util/collection-metrics.h M be/src/util/default-path-handlers.cc A be/src/util/histogram-metric.cc M be/src/util/histogram-metric.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M be/src/util/webserver.cc M be/src/util/webserver.h 33 files changed, 620 insertions(+), 458 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13501 ) Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 05 Jun 2019 03:29:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13409 ) Change subject: IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 Gerrit-Change-Number: 13409 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 05 Jun 2019 02:39:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE This commit disallows explicitly setting the Kudu table id property for Kudu tables in a ALTER TABLE set statement. The Kudu table id property is generated when the table is created and should not be altered afterwards. This commit also extracts Kudu-related analyzing tests for table alteration, so that they can be run with or without Kudu/HMS integration enabled. Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Reviewed-on: http://gerrit.cloudera.org:8080/13465 Reviewed-by: Grant Henke Reviewed-by: Thomas Marshall Reviewed-by: Alexey Serbin Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java 3 files changed, 135 insertions(+), 126 deletions(-) Approvals: Grant Henke: Looks good to me, but someone else must approve Thomas Marshall: Looks good to me, approved Alexey Serbin: Looks good to me, but someone else must approve Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 4 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 01:13:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4412/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 01:13:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 20: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 20 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 05 Jun 2019 01:00:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4411/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 20 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 05 Jun 2019 01:00:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8425: part 1: reduce size of binaries in container
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13487 ) Change subject: IMPALA-8425: part 1: reduce size of binaries in container .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13487/4/docker/impala_base/Dockerfile File docker/impala_base/Dockerfile: http://gerrit.cloudera.org:8080/#/c/13487/4/docker/impala_base/Dockerfile@30 PS4, Line 30: # Copy build artifacts required for the daemon processes. : # Need to have multiple copy commands to preserve directory structure. : COPY lib /opt/impala/lib : COPY www /opt/impala/www : COPY bin /opt/impala/bin : # Symlink here instead of in setup_build_context to avoid duplicate binaries. : RUN cd /opt/impala/bin && ln -s impalad statestored && ln -s impalad catalogd && \ : # Create conf directory for later config injection. : mkdir /opt/impala/conf && \ : # Create logs directory to collect container logs. : mkdir /opt/impala/logs > Curious: does it matter whether this is before or after the USER impala set Yeah it does - after USER impala, the commands are run as the Impala user, which doesn't have permissions to change ownership of root-owned directories. -- To view, visit http://gerrit.cloudera.org:8080/13487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ff479bedd3b93e6569e72f03f42acd9dba8b14 Gerrit-Change-Number: 13487 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:46:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8425: part 1: reduce size of binaries in container
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13487 ) Change subject: IMPALA-8425: part 1: reduce size of binaries in container .. Patch Set 4: For our test runs we would still have the unstripped binary available (I guess I hadn't confirmed that worked correctly). I don't know about the more general question of whether we want to optionally provide containers with the symbols built-in. It does seem like a lot of overhead to the size of the containers, and you would still have to extract the binary from the container to use it with GDB and a core file. It's nice that that would be an option though. We still should have enough symbols to get backtraces as-is. -- To view, visit http://gerrit.cloudera.org:8080/13487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ff479bedd3b93e6569e72f03f42acd9dba8b14 Gerrit-Change-Number: 13487 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:45:16 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-8425: Prototype to skip unneeded Kudu binaries for docker image
Joe McDonnell has abandoned this change. ( http://gerrit.cloudera.org:8080/13488 ) Change subject: WIP IMPALA-8425: Prototype to skip unneeded Kudu binaries for docker image .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/13488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I73efec0e8c1cceefd70c71f95fad8c1677e75e72 Gerrit-Change-Number: 13488 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8425: part 1: reduce size of binaries in container
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13487 ) Change subject: IMPALA-8425: part 1: reduce size of binaries in container .. Patch Set 4: (1 comment) This makes sense to me. Thanks for incorporating the Kudu change. >From a testing perspective, do we want to strip debug symbols? These tests >shouldn't crash all that often, but when they do, symbols are nice. http://gerrit.cloudera.org:8080/#/c/13487/4/docker/impala_base/Dockerfile File docker/impala_base/Dockerfile: http://gerrit.cloudera.org:8080/#/c/13487/4/docker/impala_base/Dockerfile@30 PS4, Line 30: # Copy build artifacts required for the daemon processes. : # Need to have multiple copy commands to preserve directory structure. : COPY lib /opt/impala/lib : COPY www /opt/impala/www : COPY bin /opt/impala/bin : # Symlink here instead of in setup_build_context to avoid duplicate binaries. : RUN cd /opt/impala/bin && ln -s impalad statestored && ln -s impalad catalogd && \ : # Create conf directory for later config injection. : mkdir /opt/impala/conf && \ : # Create logs directory to collect container logs. : mkdir /opt/impala/logs Curious: does it matter whether this is before or after the USER impala setting? -- To view, visit http://gerrit.cloudera.org:8080/13487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ff479bedd3b93e6569e72f03f42acd9dba8b14 Gerrit-Change-Number: 13487 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:34:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 5: I don't want to belabor this review, but I'm going to think about this overnight before commenting tomorrow. While this is clearly an effective solution, I have a gut feeling that importing ImpalaTestClusterProperties into conftest.py, and making properties like is_remote_cluster() and others available to tests from there would be a more common pattern as far as pytest is concerned. I realize that this issue has been languishing though, and holding up testing on deployed clusters, so I won't think about it too long. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:28:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4410/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 8 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 05 Jun 2019 00:27:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 6: Filed IMPALA-8625 -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:18:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 5: The previous failure was related to catalog V2 - I had made it the default for the dockerised tests, but then reverted that change. It looks like the precommit hit a crash this time around. Unfortunately minimal diagnostics. # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7f0f139bc6e0, pid=11, tid=0x7f0ba23b1700 # # JRE version: OpenJDK Runtime Environment (8.0_212-b03) (build 1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03) # Java VM: OpenJDK 64-Bit Server VM (25.212-b03 mixed mode linux-amd64 compressed oops) # Problematic frame: # C 0x7f0f139bc6e0 # # Core dump written. Default location: /opt/impala/core or core.11 # # An error report file with more information is saved as: # /tmp/hs_err_pid11.log -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:08:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 05 Jun 2019 00:09:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4409/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Jun 2019 00:07:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 5: > Patch Set 4: > > Yeah that was caused by https://issues.apache.org/jira/browse/IMPALA-8567 Hmm -- I wonder if catalog inconsistency could have any bearing IMPALA-8556 or IMPALA-8558? Is catalog_v2 the default now? -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 23:51:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4404/ -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 7 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 23:28:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 19: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 19 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 04 Jun 2019 23:24:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4402/ -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 22:59:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13507 ) Change subject: IMPALA-8450: Add support for zstd in parquet .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3510/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7 Gerrit-Change-Number: 13507 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 04 Jun 2019 22:34:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Add option to set driver in ImpalaJdbcDriver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13514 ) Change subject: Add option to set driver in ImpalaJdbcDriver .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3509/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Gerrit-Change-Number: 13514 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Jun 2019 22:25:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13504 ) Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 22:10:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13504 ) Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. IMPALA-8567: revert dockerised cluster to catalog v1 Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Reviewed-on: http://gerrit.cloudera.org:8080/13504 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M docker/catalogd/Dockerfile M docker/coord_exec/Dockerfile M docker/coordinator/Dockerfile 3 files changed, 3 insertions(+), 3 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13501 ) Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4408/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 21:57:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. IMPALA-8578: part 1: reduce dependencies on *metrics.h Before this patch there were 100s of compilation units that pulled in metrics.h and the significant amount of code in that header. It was painfully slow to recompile after changes to that file. The patch reduces that significantly and mostly eliminates transitive inclusions via other headers. * Add metrics-fwds.h with forward declarations needed to have pointers to the various classes. * Update headers to use metrics-fwds.h and move includes of *metrics.h to the .cc files. * Add includes, etc to fix compilation errors where files depended on transitively-included headers from *metrics.h This shaved about 30s off the build time on Jenkins - about a 4% speedup. I didn't end up removing anything from the headers - that is a bit more work since most of the classes are templatized and need to be explicitly instantiated in .cc files if functions are not all defined in the headers. Testing: Ran a core build Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Reviewed-on: http://gerrit.cloudera.org:8080/13491 Reviewed-by: Impala Public Jenkins Tested-by: Tim Armstrong --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/exec-node.h M be/src/exec/hash-table.h M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/plan-root-sink.cc M be/src/exec/scan-node.cc M be/src/experiments/data-provider-test.cc M be/src/experiments/tuple-splitter-test.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/reservation-tracker.cc M be/src/runtime/client-cache.cc M be/src/runtime/client-cache.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/initial-reservations.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/handle-cache.h M be/src/runtime/io/handle-cache.inline.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/local-file-reader.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/mem-pool.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-state.cc M be/src/runtime/sorter.cc M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/request-pool-service.h M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/collection-metrics.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h A be/src/util/metrics-fwd.h M be/src/util/metrics.h 60 files changed, 160 insertions(+), 61 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13501 ) Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 21:57:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 21:56:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 7: Carrying verification through rebase. -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 21:56:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 21:54:48 + Gerrit-HasComments: No
[Impala-ASF-CR] Add option to set driver in ImpalaJdbcDriver
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13514 Change subject: Add option to set driver in ImpalaJdbcDriver .. Add option to set driver in ImpalaJdbcDriver ImpalaJdbcDriver is a simple class that provides a wrapper around the Hive Jdbc driver for use in testing. This patch adds a '-d' parameter that takes a driver class and executes queries with that driver instead of the Hive driver, if its in the classpath. This makes it easy to use ./bin/run-jdbc-client.sh to test other Jdbc drivers, such as the proprietary Cloudera Impala driver. Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee --- M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java 1 file changed, 25 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/13514/1 -- To view, visit http://gerrit.cloudera.org:8080/13514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie6c8cec2f61e76b7c8321c954eef830778d8a8ee Gerrit-Change-Number: 13514 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13507 ) Change subject: IMPALA-8450: Add support for zstd in parquet .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3508/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7 Gerrit-Change-Number: 13507 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 04 Jun 2019 21:36:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13409 ) Change subject: IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4407/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 Gerrit-Change-Number: 13409 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 21:01:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Abhishek Rawat has uploaded a new patch set (#5). (
http://gerrit.cloudera.org:8080/13507 )
Change subject: IMPALA-8450: Add support for zstd in parquet
..
IMPALA-8450: Add support for zstd in parquet
Makefile was updated to include zstd in the ${IMPALA_HOME}/toolchain
directory. Other changes were made to make zstd headers and libs
accessible.
Class ZstandardCompressor/ZstandardDecompressor was added to provide
interfaces for calling ZSTD_compress/ZSTD_decompress functions. Zstd
supports different compression levels (clevel) from 1 to
ZSTD_maxCLevel(). Zstd also supports -ive clevels, but since the -ive
values represents uncompressed data they won't be supported. The default
clevel is ZSTD_CLEVEL_DEFAULT.
HdfsParquetTableWriter was updated to support ZSTD codec. The
new codecs can be set using existing query option as follows:
set COMPRESSION_CODEC=ZSTD:;
set COMPRESSION_CODEC=ZSTD; // uses ZSTD_CLEVEL_DEFAULT
Testing:
- Added unit test in DecompressorTest class with ZSTD_CLEVEL_DEFAULT
clevel and a random clevel. The test unit decompresses an input
compressed data and validates the result. It also tests for
expected behavior when passing an over/under sized buffer for
decompressing.
- Added unit tests for valid/invalid values for COMPRESSION_CODEC.
- Added e2e test in test_insert_parquet.py which tests writing/read-
ing (null/non-null) data into/from a table (w different data type
columns) using multiple codecs. Other existing e2e tests were
updated to also use parquet/zstd table format.
- Manual interoperability tests were run between Impala and Hive.
Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/experiments/compression-test.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/compress.cc
M be/src/util/compress.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M be/src/util/decompress.h
M be/src/util/runtime-profile.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
A cmake_modules/FindZstd.cmake
M common/thrift/CatalogObjects.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_error_codes.py
A
testdata/workloads/functional-query/queries/QueryTest/insert_parquet_multi_codecs.test
M tests/common/test_dimensions.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
28 files changed, 474 insertions(+), 95 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13507/5
--
To view, visit http://gerrit.cloudera.org:8080/13507
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
Gerrit-Change-Number: 13507
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3507/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 20:48:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 20:19:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13507 )
Change subject: IMPALA-8450: Add support for zstd in parquet
..
Patch Set 4:
(4 comments)
Minor comments only then I can +2
http://gerrit.cloudera.org:8080/#/c/13507/4/be/src/service/query-options-test.cc
File be/src/service/query-options-test.cc:
http://gerrit.cloudera.org:8080/#/c/13507/4/be/src/service/query-options-test.cc@23
PS4, Line 23: #include
This should be surrounded by double quotes and in the lower block, since gutil
is just part of our codebase at this point.
http://gerrit.cloudera.org:8080/#/c/13507/4/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/13507/4/be/src/service/query-options.cc@283
PS4, Line 283: if (tokens.size() > 2) { return Status("Invalid
compression codec value"); }
Remove braces for one line if
http://gerrit.cloudera.org:8080/#/c/13507/4/be/src/service/query-options.cc@285
PS4, Line 285: string codec_name = tokens[0];
nit: could be const string&
http://gerrit.cloudera.org:8080/#/c/13507/4/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/13507/4/tests/query_test/test_insert_parquet.py@150
PS4, Line 150: base_table = "%s.%s" % (unique_database, "t1_default")
nit: prefer "{0}.{1}".format(...) - it's considered a somewhat better practice
than the % operator.
--
To view, visit http://gerrit.cloudera.org:8080/13507
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
Gerrit-Change-Number: 13507
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 04 Jun 2019 20:18:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 20:17:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Michael Ho has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges As shown in IMPALA-8561, there are some paths in the code which create uncacheable ScanRanges. These uncacheable ScanRanges have mtime of -1. 'mtime' is used for differentiating versions of files with the same names. An mtime == -1 means the cache entry could potentially be from any versions of a file with the same name. This change skips lookup or insertion of ScanRange with negative mtime, file offset or buffer length. Testing done: Added targeted test cases in data-cache-test Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 --- M be/src/runtime/io/data-cache-test.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/data-cache.h M be/src/runtime/io/request-ranges.h 4 files changed, 47 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/13369/3 -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13369/1/be/src/runtime/io/data-cache-test.cc File be/src/runtime/io/data-cache-test.cc: http://gerrit.cloudera.org:8080/#/c/13369/1/be/src/runtime/io/data-cache-test.cc@267 PS1, Line 267: // Test with uncacheable 'mtime' to make sure the entry is not stored. > I actually meant those should be additional test cases, i.e. test -1 (the m Added one more test case for the uncacheable scan ranges in the latest PS. -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 20:08:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4406/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 19:49:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 19:49:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13409 ) Change subject: IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3506/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 Gerrit-Change-Number: 13409 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 19:38:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 19:11:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13409 ) Change subject: IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 Gerrit-Change-Number: 13409 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 19:10:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/13409 ) Change subject: IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/13409/2/testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test File testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test: http://gerrit.cloudera.org:8080/#/c/13409/2/testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test@10 PS2, Line 10: # Alter master address to a different location > Is this test relevant to HMS alter? Done http://gerrit.cloudera.org:8080/#/c/13409/2/testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test@60 PS2, Line 60: QUERY > I am not sure I understand the relevance of all these range partitioning ch I think it is nice to ensure alter a table to add /drop range partitions is still working with HMS integration enabled. http://gerrit.cloudera.org:8080/#/c/13409/2/testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test@274 PS2, Line 274: # Add a row > The tests altering columns make sense. ACK -- To view, visit http://gerrit.cloudera.org:8080/13409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 Gerrit-Change-Number: 13409 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:56:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration
Hello Thomas Marshall, Alexey Serbin, Andrew Wong, Grant Henke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13409 to look at the new patch set (#3). Change subject: IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration .. IMPALA-8506: Support RENAME TABLE statement with Kudu/HMS integration This commit intends to support the actual handling of ALTER/RENAME TABLE DDL for managed Kudu tables with Kudu's integration with the Hive Metastore. However, currently Kudu is considered as the source of truth of the table schema, so when ALTER TABLE (ADD/DROP COLUMN/RANGE_PARTITION), Impala always directly alters/loads the Kudu tables. Thus, this commit only updates RENAME TABLE DDL, so that after the table is renamed in the Kudu, relies on Kudu to rename the table in the HMS. Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 4 files changed, 660 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/13409/3 -- To view, visit http://gerrit.cloudera.org:8080/13409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If7155e0b385b8ad81eda0a84277bc85171a88269 Gerrit-Change-Number: 13409 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3503/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 18:41:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13501 ) Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3504/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 18:38:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3505/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3502/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 7 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:33:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4405/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:20:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java File fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java: http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@118 PS2, Line 118: // Checking for 'EXTERNAL' is case-insensitive, see IMPALA-5637. : String keyForExternalProperty = : MetaStoreUtil.findTblPropKeyCaseInsensitive(tblProperties_, "EXTERNAL"); > nit: while you are here, maybe move this under the 'if (authzConfig.isEnabl Done http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@132 PS2, Line 132: manually > nit: what does 'manually' means in this context? Is it possible to set it Done http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@132 PS2, Line 132: > nit: drop the extra space or drop the space and the period altogether. Done -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:19:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Hello Thomas Marshall, Alexey Serbin, Andrew Wong, Grant Henke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13465 to look at the new patch set (#3). Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE This commit disallows explicitly setting the Kudu table id property for Kudu tables in a ALTER TABLE set statement. The Kudu table id property is generated when the table is created and should not be altered afterwards. This commit also extracts Kudu-related analyzing tests for table alteration, so that they can be run with or without Kudu/HMS integration enabled. Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java 3 files changed, 135 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13465/3 -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 3 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3501/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 6 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:11:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 7 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 18:06:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4404/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 7 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:57:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Hello Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13501 to look at the new patch set (#4). Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. IMPALA-8578: part 2: move metrics code to .cc files This moves a lot of metric function definitions into .cc files, to reduce the size of compilation units and to reduce the frequency of recompilation required when making changes to metrics. This moves most of the large, non-perf-critical metric functions into .cc files. For template classes, this requires explicitly instantiating all combinations of template parameters that are used in impala, including in tests. Disable weak-template-vtables warning because of spurious warnings on template instantiations. See https://bugs.llvm.org/show_bug.cgi?id=18733 Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 --- M .clang-tidy M be/src/exprs/expr-test.cc M be/src/rpc/rpc-trace.cc M be/src/rpc/rpc-trace.h M be/src/rpc/thrift-server.cc M be/src/runtime/bufferpool/reservation-tracker-test.cc M be/src/runtime/coordinator.cc M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/runtime-filter-bank.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/CMakeLists.txt M be/src/util/auth-util.cc A be/src/util/collection-metrics.cc M be/src/util/collection-metrics.h M be/src/util/default-path-handlers.cc A be/src/util/histogram-metric.cc M be/src/util/histogram-metric.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M be/src/util/webserver.cc M be/src/util/webserver.h 33 files changed, 620 insertions(+), 458 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13501/4 -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13491 to look at the new patch set (#7). Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. IMPALA-8578: part 1: reduce dependencies on *metrics.h Before this patch there were 100s of compilation units that pulled in metrics.h and the significant amount of code in that header. It was painfully slow to recompile after changes to that file. The patch reduces that significantly and mostly eliminates transitive inclusions via other headers. * Add metrics-fwds.h with forward declarations needed to have pointers to the various classes. * Update headers to use metrics-fwds.h and move includes of *metrics.h to the .cc files. * Add includes, etc to fix compilation errors where files depended on transitively-included headers from *metrics.h This shaved about 30s off the build time on Jenkins - about a 4% speedup. I didn't end up removing anything from the headers - that is a bit more work since most of the classes are templatized and need to be explicitly instantiated in .cc files if functions are not all defined in the headers. Testing: Ran a core build Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/exec-node.h M be/src/exec/hash-table.h M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/plan-root-sink.cc M be/src/exec/scan-node.cc M be/src/experiments/data-provider-test.cc M be/src/experiments/tuple-splitter-test.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/reservation-tracker.cc M be/src/runtime/client-cache.cc M be/src/runtime/client-cache.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/initial-reservations.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/handle-cache.h M be/src/runtime/io/handle-cache.inline.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/local-file-reader.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/mem-pool.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-state.cc M be/src/runtime/sorter.cc M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/request-pool-service.h M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/collection-metrics.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h A be/src/util/metrics-fwd.h M be/src/util/metrics.h 60 files changed, 160 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/13491/7 -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Hello Thomas Marshall, Alexey Serbin, Andrew Wong, Grant Henke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13400 to look at the new patch set (#7). Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration This commit supports the actual handling of DROP TABLE DDL for managed Kudu tables with Kudu's integration with the Hive Metastore. When Kudu/HMS integration is enabled, after the table is dropped in the Kudu, relies on Kudu to drop the table in the HMS. Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_kudu.py 2 files changed, 149 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/13400/7 -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 7 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Thomas Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration This commit supports the actual handling of CREATE TABLE DDL for managed Kudu tables when integration with Hive Metastore is enabled. When Kudu/HMS integration is enabled, for CREATE TABLE statement, Impala can rely on Kudu to create the table in the HMS. Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Reviewed-on: http://gerrit.cloudera.org:8080/13375 Reviewed-by: Thomas Marshall Reviewed-by: Grant Henke Tested-by: Thomas Marshall --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M tests/common/custom_cluster_test_suite.py M tests/common/kudu_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 10 files changed, 306 insertions(+), 20 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved; Verified Grant Henke: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 10 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 9: Verified+1 There haven't been any code changes since the verify job on patch set 7, so I'll go ahead and verify it (the more recent verify failure was because I aborted the job) -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 9 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:36:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 9: -Code-Review Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3500/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 9 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:33:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 9: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 9 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:33:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 6 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:33:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Hello Thomas Marshall, Alexey Serbin, Andrew Wong, Grant Henke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13400 to look at the new patch set (#6). Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration This commit supports the actual handling of DROP TABLE DDL for managed Kudu tables with Kudu's integration with the Hive Metastore. When Kudu/HMS integration is enabled, after the table is dropped in the Kudu, relies on Kudu to drop the table in the HMS. Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_kudu.py 2 files changed, 149 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/13400/6 -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 6 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/13400/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13400/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1590 PS5, Line 1590: KuduTable.isKuduTable(msTbl) && !Table.isExternalTable(msTbl); > nit: Maybe separate setting isManagedKuduTable from the drop. Done http://gerrit.cloudera.org:8080/#/c/13400/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1943 PS5, Line 1943: // the HMS table for managed tables. > Nit, can you add comment here too. Done -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 6 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:31:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 8 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:31:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Hello Thomas Marshall, Alexey Serbin, Andrew Wong, Grant Henke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13375 to look at the new patch set (#9). Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration This commit supports the actual handling of CREATE TABLE DDL for managed Kudu tables when integration with Hive Metastore is enabled. When Kudu/HMS integration is enabled, for CREATE TABLE statement, Impala can rely on Kudu to create the table in the HMS. Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M tests/common/custom_cluster_test_suite.py M tests/common/kudu_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 10 files changed, 306 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/13375/9 -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 9 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 8 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:30:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4403/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 8 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 17:31:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4402/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 17:26:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3499/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 17:09:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13504 ) Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3498/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 17:05:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13507 ) Change subject: IMPALA-8450: Add support for zstd in parquet .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3497/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7 Gerrit-Change-Number: 13507 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 04 Jun 2019 16:55:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Alexey Serbin has posted comments on this change. (
http://gerrit.cloudera.org:8080/13375 )
Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with
Kudu/HMS integration
..
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13375/4/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
File fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java:
http://gerrit.cloudera.org:8080/#/c/13375/4/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java@120
PS4, Line 120:* Return the value of thrift URI for the remote Hive
Metastore.
:*/
: public static String
getHiveMetastoreUrisKeyValue(IMetaStoreClient client)
: throws ConfigValSecurityException, TException {
: return client.getConfigValue(
: HIVE_METASTORE_URIS_KEY, DEFAULT_HIVE_METASTORE_URIS);
: }
:
> This is to ensure always use the correct key (HIVE_METASTORE_URIS_KEY) when
If the constant HIVE_METASTORE_URIS_KEY is used, then I don't see much risk
specifying a wrong key accidentally. However, if I'm OK with keeping it since
it was just a nit comment, actually.
--
To view, visit http://gerrit.cloudera.org:8080/13375
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502
Gerrit-Change-Number: 13375
Gerrit-PatchSet: 7
Gerrit-Owner: Hao Hao
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Comment-Date: Tue, 04 Jun 2019 16:50:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java File fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java: http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@118 PS2, Line 118: // Checking for 'EXTERNAL' is case-insensitive, see IMPALA-5637. : String keyForExternalProperty = : MetaStoreUtil.findTblPropKeyCaseInsensitive(tblProperties_, "EXTERNAL"); nit: while you are here, maybe move this under the 'if (authzConfig.isEnabled())' clause? http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@132 PS2, Line 132: manually nit: what does 'manually' means in this context? Is it possible to set it 'programmatically'? I would rather drop this 'manually' to avoid ambiguity. Maybe, replace the sentence: "property '%s' cannot be altered for Kudu tables" http://gerrit.cloudera.org:8080/#/c/13465/2/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@132 PS2, Line 132: nit: drop the extra space or drop the space and the period altogether. -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 04 Jun 2019 16:44:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13369 ) Change subject: IMPALA-8562: Data cache should skip insertion of uncacheable ScanRanges .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13369/1/be/src/runtime/io/data-cache-test.cc File be/src/runtime/io/data-cache-test.cc: http://gerrit.cloudera.org:8080/#/c/13369/1/be/src/runtime/io/data-cache-test.cc@267 PS1, Line 267: ASSERT_FALSE(cache.Store(FNAME, -1000, 0, test_buffer(), TEMP_BUFFER_SIZE)); > Done I actually meant those should be additional test cases, i.e. test -1 (the most common value) and some other arbitrary out-of-range value since it's cheap to add additional tests here. -- To view, visit http://gerrit.cloudera.org:8080/13369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2294833b075a2ddcae956d9fdb04f3e85adb0391 Gerrit-Change-Number: 13369 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 16:41:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13504 ) Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4401/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 16:36:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4400/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 16:32:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 16:32:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 16:32:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13504 ) Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 16:31:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13491 ) Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/13491/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13491/4//COMMIT_MSG@11 PS4, Line 11: painfuly > nit: painfully Done http://gerrit.cloudera.org:8080/#/c/13491/4/be/src/rpc/TAcceptQueueServer.h File be/src/rpc/TAcceptQueueServer.h: http://gerrit.cloudera.org:8080/#/c/13491/4/be/src/rpc/TAcceptQueueServer.h@78 PS4, Line 78: std:: > It's strange that it compiles without the namespace-qualifier. Turned out w Yeah I don't think it's a major issue but it's nice to stay consistent about using std:: in headers. http://gerrit.cloudera.org:8080/#/c/13491/4/be/src/util/metric-defs.h File be/src/util/metric-defs.h: http://gerrit.cloudera.org:8080/#/c/13491/4/be/src/util/metric-defs.h@1 PS4, Line 1: // Licensed to the Apache Software Foundation (ASF) under one > nit: for filename, maybe 'metric-fwd.h', like ? Went with metrics-fwd. -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 16:30:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8578: part 1: reduce dependencies on *metrics.h
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13491 to look at the new patch set (#5). Change subject: IMPALA-8578: part 1: reduce dependencies on *metrics.h .. IMPALA-8578: part 1: reduce dependencies on *metrics.h Before this patch there were 100s of compilation units that pulled in metrics.h and the significant amount of code in that header. It was painfully slow to recompile after changes to that file. The patch reduces that significantly and mostly eliminates transitive inclusions via other headers. * Add metrics-fwds.h with forward declarations needed to have pointers to the various classes. * Update headers to use metrics-fwds.h and move includes of *metrics.h to the .cc files. * Add includes, etc to fix compilation errors where files depended on transitively-included headers from *metrics.h This shaved about 30s off the build time on Jenkins - about a 4% speedup. I didn't end up removing anything from the headers - that is a bit more work since most of the classes are templatized and need to be explicitly instantiated in .cc files if functions are not all defined in the headers. Testing: Ran a core build Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/exec-node.h M be/src/exec/hash-table.h M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/plan-root-sink.cc M be/src/exec/scan-node.cc M be/src/experiments/data-provider-test.cc M be/src/experiments/tuple-splitter-test.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/reservation-tracker.cc M be/src/runtime/client-cache.cc M be/src/runtime/client-cache.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/initial-reservations.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/handle-cache.h M be/src/runtime/io/handle-cache.inline.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/local-file-reader.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/mem-pool.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-state.cc M be/src/runtime/sorter.cc M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/request-pool-service.h M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/collection-metrics.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h A be/src/util/metrics-fwd.h M be/src/util/metrics.h 60 files changed, 160 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/13491/5 -- To view, visit http://gerrit.cloudera.org:8080/13491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie2942366cab5421f2db7c27e7da712ea6f775fdb Gerrit-Change-Number: 13491 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13504 ) Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. Patch Set 2: The dockerised build hit a build error that looked unrelated. I reran and it succeeded. https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/436/ -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Jun 2019 16:26:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8567: revert dockerised cluster to catalog v1
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13504 Change subject: IMPALA-8567: revert dockerised cluster to catalog v1 .. IMPALA-8567: revert dockerised cluster to catalog v1 Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c --- M docker/catalogd/Dockerfile M docker/coord_exec/Dockerfile M docker/coordinator/Dockerfile 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/13504/2 -- To view, visit http://gerrit.cloudera.org:8080/13504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icf60b7ed7a22cc176d68ded1da23e4445750097c Gerrit-Change-Number: 13504 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8450: Add support for zstd in parquet
Abhishek Rawat has uploaded a new patch set (#4). (
http://gerrit.cloudera.org:8080/13507 )
Change subject: IMPALA-8450: Add support for zstd in parquet
..
IMPALA-8450: Add support for zstd in parquet
Makefile was updated to include zstd in the ${IMPALA_HOME}/toolchain
directory. Other changes were made to make zstd headers and libs
accessible.
Class ZstandardCompressor/ZstandardDecompressor was added to provide
interfaces for calling ZSTD_compress/ZSTD_decompress functions. Zstd
supports different compression levels (clevel) from 1 to
ZSTD_maxCLevel(). Zstd also supports -ive clevels, but since the -ive
values represents uncompressed data they won't be supported. The default
clevel is ZSTD_CLEVEL_DEFAULT.
HdfsParquetTableWriter was updated to support ZSTD codec. The
new codecs can be set using existing query option as follows:
set COMPRESSION_CODEC=ZSTD:;
set COMPRESSION_CODEC=ZSTD; // uses ZSTD_CLEVEL_DEFAULT
Testing:
- Added unit test in DecompressorTest class with ZSTD_CLEVEL_DEFAULT
clevel and a random clevel. The test unit decompresses an input
compressed data and validates the result. It also tests for
expected behavior when passing an over/under sized buffer for
decompressing.
- Added unit tests for valid/invalid values for COMPRESSION_CODEC.
- Added e2e test in test_insert_parquet.py which tests writing/read-
ing (null/non-null) data into/from a table (w different data type
columns) using multiple codecs. Other existing e2e tests were
updated to also use parquet/zstd table format.
- Manual interoperability tests were run between Impala and Hive.
Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/experiments/compression-test.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/compress.cc
M be/src/util/compress.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M be/src/util/decompress.h
M be/src/util/runtime-profile.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
A cmake_modules/FindZstd.cmake
M common/thrift/CatalogObjects.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_error_codes.py
A
testdata/workloads/functional-query/queries/QueryTest/insert_parquet_multi_codecs.test
M tests/common/test_dimensions.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
28 files changed, 474 insertions(+), 95 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13507/4
--
To view, visit http://gerrit.cloudera.org:8080/13507
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
Gerrit-Change-Number: 13507
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8578: part 2: move metrics code to .cc files
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13501 ) Change subject: IMPALA-8578: part 2: move metrics code to .cc files .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78ad045ded6e6a7b7524711be9302c26115b97b9 Gerrit-Change-Number: 13501 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 14:22:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3496/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 14:09:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13508/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13508/1//COMMIT_MSG@7 PS1, Line 7: NPE nit: might worth spell it out at least once. http://gerrit.cloudera.org:8080/#/c/13508/1/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/1/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@124 PS1, Line 124: analyzer.addAccessEvent(new TAccessEvent( Should we log the exception? -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jun 2019 13:56:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13508 Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. IMPALA-8612: Fix sporadic NPE when dropping an authorized table In the analyze() function of DropTableOrViewStmt it's possible that serverName_ is not set when analyzer.getTable() throws. As a result when the Catalog executes the drop table DDL it runs into a failing Precondition check and throws a NPE when updating user privileges. Note, to run into the NPE it's required to have authorization enabled. Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 --- M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/13508/1 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
