[Impala-ASF-CR] Add missing authorization in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 Aug 2018 04:06:07 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Add missing authorization in KRPC In 2.12.0, Impala adopted Kudu RPC library for certain backened services (TransmitData(), EndDataStream()). While the implementation uses Kerberos for authenticating users connecting to the backend services, there is no authorization implemented. This is a regression from the Thrift based implementation because it registered a SASL callback (SaslAuthorizeInternal) to be invoked during the connection negotiation. With this regression, an unauthorized but authenticated user may invoke RPC calls to Impala backend services. This change fixes the issue above by overriding the default authorization method for the DataStreamService. The authorization method will only let authenticated principal which matches FLAGS_principal / FLAGS_be_principal to access the service. Also added a new startup flag --krb5_ccname to allow users to customize the locations of the Kerberos credentials cache. Testing done: 1. Added a new test case in rpc-mgr-kerberized-test.cc to confirm an unauthorized user is not allowed to access the service. 2. Ran some queries in a Kerberos enabled cluster to make sure there is no error. 3. Exhaustive builds. Thanks to Todd Lipcon for pointing out the problem and his guidance on the fix. Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Reviewed-on: http://gerrit.cloudera.org:8080/11331 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/rpc/authentication.cc M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc R be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/thrift-server-test.cc M be/src/runtime/data-stream-test.cc M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/testutil/mini-kdc-wrapper.cc M be/src/testutil/mini-kdc-wrapper.h M be/src/util/auth-util.cc M be/src/util/auth-util.h M bin/rat_exclude_files.txt M common/protobuf/data_stream_service.proto A common/protobuf/kudu M common/protobuf/rpc_test.proto 19 files changed, 332 insertions(+), 135 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7425: Change incremental stats to pull from catalogd.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11193 ) Change subject: IMPALA-7425: Change incremental stats to pull from catalogd. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/533/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11193 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d564808ca5157afe4e091909ca6cdac76e60d6e Gerrit-Change-Number: 11193 Gerrit-PatchSet: 8 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 30 Aug 2018 03:50:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7425: Change incremental stats to pull from catalogd.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11193 ) Change subject: IMPALA-7425: Change incremental stats to pull from catalogd. .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/11193/8/tests/custom_cluster/test_pull_stats.py File tests/custom_cluster/test_pull_stats.py: http://gerrit.cloudera.org:8080/#/c/11193/8/tests/custom_cluster/test_pull_stats.py@21 PS8, Line 21: class TestPullStatistics(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/11193 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d564808ca5157afe4e091909ca6cdac76e60d6e Gerrit-Change-Number: 11193 Gerrit-PatchSet: 8 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 30 Aug 2018 03:07:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7425: Change incremental stats to pull from catalogd.
Hello Bharath Vissapragada, Tianyi Wang, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11193 to look at the new patch set (#8). Change subject: IMPALA-7425: Change incremental stats to pull from catalogd. .. IMPALA-7425: Change incremental stats to pull from catalogd. Currently, incremental stats can consume a substantial amount of metadata memory (per table, partition, column). This metadata is transmitted from catalogd to all coordinators. As a result, memory is used for all loaded tables that use incremental stats all the time at all coordinators. A consequence is that coordinators and catalogd die from OOM more often when incremental stats are used and more network bandwidth is used. This change removes incremental stats from impalads. These stats are only needed when computing incremental statistics and merging new results with the existing results. They are not used by queries. As a result, the change requires that coordinators fetch incremental stats directly from catalogd when computing incremental stats. In addition, catalogd no longer sends incremental stats to coordinators via the statestore. The option is enabled by setting a new flag, --pull_incremental_statistics, on the catalogd and all impalad coordinators. Testing: - manual testing - added end-to-end tests with --pull_incremental_statistics enabled for the compute-stats-incremental.test - added fe CatalogTest for new catalogd service method - passes exhaustive tests when --pull_incremental_statistics is enabled and disabled Change-Id: I9d564808ca5157afe4e091909ca6cdac76e60d6e --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/common/global-flags.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/common/custom_cluster_test_suite.py M tests/conftest.py A tests/custom_cluster/test_pull_stats.py 26 files changed, 634 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/11193/8 -- To view, visit http://gerrit.cloudera.org:8080/11193 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9d564808ca5157afe4e091909ca6cdac76e60d6e Gerrit-Change-Number: 11193 Gerrit-PatchSet: 8 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7425: Change incremental stats to pull from catalogd.
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/11193 )
Change subject: IMPALA-7425: Change incremental stats to pull from catalogd.
..
Patch Set 4:
(11 comments)
main change here is to use names for partitions instead of the ids.
http://gerrit.cloudera.org:8080/#/c/11193/4/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/11193/4/be/src/catalog/catalog-server.cc@149
PS4, Line 149: ;
> Can you log the request too? Contains table info etc
the java fn logs this info
http://gerrit.cloudera.org:8080/#/c/11193/4/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/11193/4/be/src/common/global-flags.cc@210
PS4, Line 210: If used, the flag must "
: "be set on both catalogd and all impalad coordinators.");
> Since this is for v1 metadata, I think it would be even cleaner if it was s
backing off on this change to keep the scope of the change down. added a todo.
http://gerrit.cloudera.org:8080/#/c/11193/4/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/11193/4/common/thrift/CatalogService.thrift@380
PS4, Line 380: // identified by their id. The caller's table_version_number is
used by the catalog
yes, I considered doing the work on the catalogd side (shift the merge from
coordinator to catalog) but opted to stick with a dataflow that's closer to
what's currently there (e.g., primary diff is to fetch remote instead of
local... all the other plumbing remains the same and we don't run the c++ code
for merging in catalog).
updated revision goes with partitioning values as more stable partition names.
the scheme is for catalogd to send everything then let the impalad remove whats
needed, which is expected to be small (so we're usually sending most partition
stats). the issue with versions is avoided, but on the impalad, we do detect if
partitions have been added/removed and fail if we see we're out-of-sync in this
regard.
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@452
PS7, Line 452: ionStats != null)
> nit: the format for this should be /*excludedPartitions=*/ so that ErrorPr
Done
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@454
PS7, Line 454: te_col_stats.isEmpty() && !tableIsMissingColStats;
: if (!hasStat
> nit: combine
Done
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@610
PS7, Line 610: if (exclude
> nit: i think it's worth writing 'expected' instead of 'exp' since my mind w
Done
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@644
PS7, Line 644: } catch (Exception e) {
> it seems like the scoping in this function could be tightened up a bit by g
Done
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@410
PS7, Line 410: able_v
> missing the '%s' here. Side note: not sure why Impala likes to use String.f
Done. not sure. perhaps it was a habit formed when concat was considered slower
than alternatives. when I use this I'm primarily sticking to what's in the
current file. if we want to change that, I'd prefer to do change all uses per
file, but would prefer to avoid that for now to avoid conflicts.
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1742
PS7, Line 1742: ponding to
> this javadoc is a bit incomplete since it also depends how many partitions
Done
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1744
PS7, Line 1744: erwis
> nit: "shouldSend..." so that it's a question rather than a verb. To me, "ch
Done
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java:
http://gerrit.cloudera.org:8080/#/c/11193/7/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@211
PS7, Line 211: OperationException("N/A");
: }
> did you mean the opposite of this?
whoops,
[Impala-ASF-CR] IMPALA-7424: Reduce in-memory footprint of incremental stats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11341 ) Change subject: IMPALA-7424: Reduce in-memory footprint of incremental stats .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/532/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39f02ebfa0c6e9b0baedd0d76058a1b34efb5a02 Gerrit-Change-Number: 11341 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 30 Aug 2018 02:46:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [diagnostics] Make --minidump dir consistent with Impala's --minidump path
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11353 ) Change subject: [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3094/ -- To view, visit http://gerrit.cloudera.org:8080/11353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 Gerrit-Change-Number: 11353 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 30 Aug 2018 02:17:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7424: Reduce in-memory footprint of incremental stats
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11341 ) Change subject: IMPALA-7424: Reduce in-memory footprint of incremental stats .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/11341/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/11341/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@819 PS5, Line 819: p Will remove in next PS http://gerrit.cloudera.org:8080/#/c/11341/5/fe/src/test/java/org/apache/impala/util/CompressionUtilTest.java File fe/src/test/java/org/apache/impala/util/CompressionUtilTest.java: http://gerrit.cloudera.org:8080/#/c/11341/5/fe/src/test/java/org/apache/impala/util/CompressionUtilTest.java@32 PS5, Line 32: a Will update in next PS -- To view, visit http://gerrit.cloudera.org:8080/11341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39f02ebfa0c6e9b0baedd0d76058a1b34efb5a02 Gerrit-Change-Number: 11341 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 30 Aug 2018 02:09:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7424: Reduce in-memory footprint of incremental stats
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11341 ) Change subject: IMPALA-7424: Reduce in-memory footprint of incremental stats .. Patch Set 5: (29 comments) http://gerrit.cloudera.org:8080/#/c/11341/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11341/4//COMMIT_MSG@7 PS4, Line 7: Reduce i > nit: Reduce Done http://gerrit.cloudera.org:8080/#/c/11341/4//COMMIT_MSG@10 PS4, Line 10: HMS parameters map of partition objects. Each of these strings when > nit: ... when stored in catalogd. Done http://gerrit.cloudera.org:8080/#/c/11341/4//COMMIT_MSG@11 PS4, Line 11: ts th > nit: take Done http://gerrit.cloudera.org:8080/#/c/11341/4//COMMIT_MSG@17 PS4, Line 17: persist > persistent? Done http://gerrit.cloudera.org:8080/#/c/11341/4//COMMIT_MSG@26 PS4, Line 26: improves > nit: improves reminds me of my advisor suggesting edits to my thesis draft :D http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogObjects.thrift File common/thrift/CatalogObjects.thrift: http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogObjects.thrift@297 PS4, Line 297: byte[] representation of T > still having trouble understanding this one. how about: "byte[] representat Done http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogService.thrift File common/thrift/CatalogService.thrift: http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogService.thrift@283 PS4, Line 283: should i > nit: should include (for consistency) Done http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogService.thrift@284 PS4, Line 284: and that > nit: that is Done http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogService.thrift@302 PS4, Line 302: 'want_partition_stats' was set i > sync this with the field name in that struct. Done http://gerrit.cloudera.org:8080/#/c/11341/4/common/thrift/CatalogService.thrift@307 PS4, Line 307: 'want_partition_stats' in TTable > sync Done http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java: http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@a446 PS4, Line 446: > any reason to get rid of these? Didn't seem like they are any value add to debugging? http://gerrit.cloudera.org:8080/#/c/11341/3/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java: http://gerrit.cloudera.org:8080/#/c/11341/3/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@337 PS3, Line 337: > Do you recall where the '400' came from in the first place? Looking through Yep, it was an empirical estimate based on some clusters with incremental stats. I'm adding a TODO to fix that. http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java: http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@334 PS4, Line 334: .setPartitio > checkNotNull returns its argument, so you can inline this into the paramete Done http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java File fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java: http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java@120 PS4, Line 120:* are stored as a deflate-compressed byte array to reduce memory footprint. Use > perhaps include the util method to use to turn it into a TPartitionStats? Done http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java@158 PS4, Line 158: > nit: remove Done http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@527 PS4, Line 527: : private boolean hasIncrementalStats_; > remove Done http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@637 PS4, Line 637: if (hasIncrStats) Preconditions.checkNotNull(partitionStats); > worth a precondition check that, if hasIncrStats is true, then partitionSta Done http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@637 PS4, Line 637: if (hasIncrStats) Preconditions.checkNotNull(partitionStats); > you mentioned on L527 that this should be set once-- perhaps enforce it? Removed. http://gerrit.cloudera.org:8080/#/c/11341/4/fe/src/main/java/org/apache/impala/catal
[Impala-ASF-CR] IMPALA-7424: Reduce in-memory footprint of incremental stats
Hello Todd Lipcon, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11341 to look at the new patch set (#5). Change subject: IMPALA-7424: Reduce in-memory footprint of incremental stats .. IMPALA-7424: Reduce in-memory footprint of incremental stats Currently incremental stats are stored as chunked Base64 strings in the HMS parameters map of partition objects. Each of these strings when stored in the catalogd are Java 'String' objects that use UTF-16 encoding and take up to 2 bytes per character. This patch converts the string representation into a gzipped byte array form when the partition is loaded in the Catalogd and this state is maintained when transmitting them to the coordinators. To maintain backward compatibility, the persistent HMS representation of stats has not been modified. So the incremental stats are still written back to the chunked Base64 representation while serializing the partition state to HMS. On a real world catalogserver dominated by incremental stats memory footprint, this patch showed ~54% end-to-end heapsize reduction and ~79% reduction in the memory footprint of incremental stats data structures. This patch also improves the way the callers check if a partition has incremental stats by computing this information once and reusing it later. Without the patch, we deserialize the entire incremental stats structure everytime this information is needed and that triggers a spike in usage of working memory on catalogds/Impalads. Testing: Ran core tests on Catalog V1 Implementation. Ran some manual queries on Catalog V2 implementation. Change-Id: I39f02ebfa0c6e9b0baedd0d76058a1b34efb5a02 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CompressionUtil.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java A fe/src/test/java/org/apache/impala/util/CompressionUtilTest.java 17 files changed, 333 insertions(+), 117 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/11341/5 -- To view, visit http://gerrit.cloudera.org:8080/11341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I39f02ebfa0c6e9b0baedd0d76058a1b34efb5a02 Gerrit-Change-Number: 11341 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7335: Fix a race in HdfsScanNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11337 ) Change subject: IMPALA-7335: Fix a race in HdfsScanNode .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/531/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id470818846a5c69ad28a6ff695069702982aa793 Gerrit-Change-Number: 11337 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 Aug 2018 01:41:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11356 ) Change subject: IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/58/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f45b42f3300507296c3a3e876498976c0b17a86 Gerrit-Change-Number: 11356 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 Aug 2018 01:18:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11356 Change subject: IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges .. IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges Change-Id: I5f45b42f3300507296c3a3e876498976c0b17a86 --- M docs/shared/impala_common.xml 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/11356/1 -- To view, visit http://gerrit.cloudera.org:8080/11356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5f45b42f3300507296c3a3e876498976c0b17a86 Gerrit-Change-Number: 11356 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11356 ) Change subject: IMPALA-7502: [DOCS] Update ALTER TABLE/VIEW RENAME privileges .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/58/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f45b42f3300507296c3a3e876498976c0b17a86 Gerrit-Change-Number: 11356 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 Aug 2018 01:15:08 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/530/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 Aug 2018 01:14:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5937: [DOCS] PARQUET READ STATISTICS and PARQUET DICTIONARY FILTERING
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11355 ) Change subject: IMPALA-5937: [DOCS] PARQUET_READ_STATISTICS and PARQUET_DICTIONARY_FILTERING .. Patch Set 2: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/57/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 Gerrit-Change-Number: 11355 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 Aug 2018 01:12:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11344 ) Change subject: IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table .. IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table Prior to this patch, ALTER TABLE/VIEW RENAME required ALTER on the old table. This may pose a potential security risk, such as having ALTER on a table and ALL on a particular database allows a user to move the table to a database with ALL, which will automatically grant that user with ALL privilege on that table due to the privilege inherited from the database. This patch fixes the issue by requring ALL on the old table. What this means is moving a table to a database with ALL privilege will not elevate the privilege since ALL is now required for a table to be renamed. Testing: - Ran all FE tests Change-Id: I47a417a77df3f3030cf5f54fd2280b5e5e1fb77a Reviewed-on: http://gerrit.cloudera.org:8080/11344 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 3 files changed, 57 insertions(+), 43 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I47a417a77df3f3030cf5f54fd2280b5e5e1fb77a Gerrit-Change-Number: 11344 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11344 ) Change subject: IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47a417a77df3f3030cf5f54fd2280b5e5e1fb77a Gerrit-Change-Number: 11344 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 30 Aug 2018 01:11:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5937: [DOCS] PARQUET READ STATISTICS and PARQUET DICTIONARY FILTERING
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11355 ) Change subject: IMPALA-5937: [DOCS] PARQUET_READ_STATISTICS and PARQUET_DICTIONARY_FILTERING .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/57/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 Gerrit-Change-Number: 11355 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 30 Aug 2018 01:06:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5937: [DOCS] PARQUET READ STATISTICS and PARQUET DICTIONARY FILTERING
Hello Lars Volker, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11355 to look at the new patch set (#2). Change subject: IMPALA-5937: [DOCS] PARQUET_READ_STATISTICS and PARQUET_DICTIONARY_FILTERING .. IMPALA-5937: [DOCS] PARQUET_READ_STATISTICS and PARQUET_DICTIONARY_FILTERING Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 --- M docs/impala.ditamap M docs/topics/impala_parquet.xml A docs/topics/impala_parquet_dictionary_filtering.xml A docs/topics/impala_parquet_read_statistics.xml 4 files changed, 219 insertions(+), 48 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/11355/2 -- To view, visit http://gerrit.cloudera.org:8080/11355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 Gerrit-Change-Number: 11355 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7335: Fix a race in HdfsScanNode
Pooja Nilangekar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11337 ) Change subject: IMPALA-7335: Fix a race in HdfsScanNode .. IMPALA-7335: Fix a race in HdfsScanNode This change fixes the race between the done_ flag and the status_ variable in the HdfsScanNode. Previously, a scanner thread would mark its scan range as complete even if it ran into an error. Another scanner thread could notice that all scan ranges have completed and set the done_ flag before the status_ variable is updated with the non-ok status. This caused the main thread to return an ok status despite the scanner thread running into an error. This change ensures that when a scanner thread runs into an error, it updates the status_ before marking its scan range as complete. Testing: Ran core tests. Since there is no deterministic method to reproduce the race, this change does not include any regression tests. Additionally, this change also fixes IMPALA-7430 by removing the logs added to debug this race. Change-Id: Id470818846a5c69ad28a6ff695069702982aa793 --- M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h 2 files changed, 39 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/11337/5 -- To view, visit http://gerrit.cloudera.org:8080/11337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id470818846a5c69ad28a6ff695069702982aa793 Gerrit-Change-Number: 11337 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7446: enable missing buffer pool GC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11350 ) Change subject: IMPALA-7446: enable missing buffer pool GC .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/529/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 Aug 2018 00:47:37 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3095/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 Aug 2018 00:32:23 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 Aug 2018 00:32:22 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 7: Code-Review+2 Carry Todd's +2 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 Aug 2018 00:31:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 7: PS7 adds 2 lines of code in rpc-mgr-kerberized-test.cc to set / clear FLAGS_krb5_ccname. -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 Aug 2018 00:31:03 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Hello Sailesh Mukil, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11331 to look at the new patch set (#7). Change subject: Add missing authorization in KRPC .. Add missing authorization in KRPC In 2.12.0, Impala adopted Kudu RPC library for certain backened services (TransmitData(), EndDataStream()). While the implementation uses Kerberos for authenticating users connecting to the backend services, there is no authorization implemented. This is a regression from the Thrift based implementation because it registered a SASL callback (SaslAuthorizeInternal) to be invoked during the connection negotiation. With this regression, an unauthorized but authenticated user may invoke RPC calls to Impala backend services. This change fixes the issue above by overriding the default authorization method for the DataStreamService. The authorization method will only let authenticated principal which matches FLAGS_principal / FLAGS_be_principal to access the service. Also added a new startup flag --krb5_ccname to allow users to customize the locations of the Kerberos credentials cache. Testing done: 1. Added a new test case in rpc-mgr-kerberized-test.cc to confirm an unauthorized user is not allowed to access the service. 2. Ran some queries in a Kerberos enabled cluster to make sure there is no error. 3. Exhaustive builds. Thanks to Todd Lipcon for pointing out the problem and his guidance on the fix. Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 --- M be/src/common/global-flags.cc M be/src/rpc/authentication.cc M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc R be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/thrift-server-test.cc M be/src/runtime/data-stream-test.cc M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/testutil/mini-kdc-wrapper.cc M be/src/testutil/mini-kdc-wrapper.h M be/src/util/auth-util.cc M be/src/util/auth-util.h M bin/rat_exclude_files.txt M common/protobuf/data_stream_service.proto A common/protobuf/kudu M common/protobuf/rpc_test.proto 19 files changed, 332 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11331/7 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-5937: [DOCS] Documented PARQUET READ STATISTICS option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11355 ) Change subject: IMPALA-5937: [DOCS] Documented PARQUET_READ_STATISTICS option .. Patch Set 1: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/56/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 Gerrit-Change-Number: 11355 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 30 Aug 2018 00:14:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7446: enable missing buffer pool GC
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11350 ) Change subject: IMPALA-7446: enable missing buffer pool GC .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc File be/src/runtime/mem-tracker-test.cc: http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc@386 PS2, Line 386: EXPECT_FALSE(t.TryConsume(1, MemLimit::SOFT)); > Is that necessary for the test? Not sure it's worth slowing down the test r good point -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 Aug 2018 00:11:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7446: enable missing buffer pool GC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11350 ) Change subject: IMPALA-7446: enable missing buffer pool GC .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/11350/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11350/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-7445 > is this the right JIRA? thanks for checking http://gerrit.cloudera.org:8080/#/c/11350/2//COMMIT_MSG@14 PS2, Line 14: new, soft, : mem_limit > nit: commas? Done http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc File be/src/runtime/mem-tracker-test.cc: http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc@384 PS2, Line 384: Most of the GC attempts should fail > nit: a bit confusing to say GC attempts fail since all GC attempts fail to Done http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc@386 PS2, Line 386: EXPECT_FALSE(t.TryConsume(1, MemLimit::SOFT)); > maybe add a 1ms sleep to ensure some on the gc attemps actually run Is that necessary for the test? Not sure it's worth slowing down the test runtime. We do test the alternative code path above. -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 Aug 2018 00:03:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7446: enable missing buffer pool GC
Hello Bikramjeet Vig, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11350 to look at the new patch set (#3). Change subject: IMPALA-7446: enable missing buffer pool GC .. IMPALA-7446: enable missing buffer pool GC This was not hooked up as an oversight, and means that sometimes queries can spill or OOM because of accumulated free buffers and pages in the buffer pool. This patch ensures that we try to free up unused memory from the buffer pool before we start failing memory allocations. I was somewhat concerned about an interaction between the soft mem_limit and the GC. If the system is near or over the soft process memory limit, it's possible that many spilling operators or HDFS scans could be repeatedly bumping up against the soft memory limit and doing failed GCs. With hard memory limits, failed GCs lead to many queries failing and the situation resolving itself, but with soft memory limits the queries will likely be able to continue running while doing many GCs. My solution was to back off soft GCs for a 100ms period after a failure. Testing: Added unit tests. Looped the query from IMPALA-7442 to confirm it didn't fail. Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 --- M be/src/common/init.cc M be/src/runtime/exec-env.cc M be/src/runtime/mem-tracker-test.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h 5 files changed, 140 insertions(+), 75 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/11350/3 -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5937: [DOCS] Documented PARQUET READ STATISTICS option
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11355 Change subject: IMPALA-5937: [DOCS] Documented PARQUET_READ_STATISTICS option .. IMPALA-5937: [DOCS] Documented PARQUET_READ_STATISTICS option Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 --- M docs/impala.ditamap A docs/topics/impala_parquet_read_statistics.xml 2 files changed, 85 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/11355/1 -- To view, visit http://gerrit.cloudera.org:8080/11355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 Gerrit-Change-Number: 11355 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-5937: [DOCS] Documented PARQUET READ STATISTICS option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11355 ) Change subject: IMPALA-5937: [DOCS] Documented PARQUET_READ_STATISTICS option .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/56/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88fa8c4a64560711251076c50e1695f7f032f9c0 Gerrit-Change-Number: 11355 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 Aug 2018 00:01:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11352 ) Change subject: IMPALA-7508: Add Impala Python GDB module .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py File infra/gdb/impala-gdb.py: http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@1 PS1, Line 1: # > I feel like this file should more properly be added to: That would be fine by me. -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 23:15:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Bump FE pom to Java 8 source/target version Our dependency on Hadoop 3 means we already required Java 8. This just fixes the pom so we are compiling to Java 8 classes. This will also allow us to start using some Java 8 features like lambdas for more concise code. Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Reviewed-on: http://gerrit.cloudera.org:8080/11351 Reviewed-by: Philip Zeyliger Tested-by: Impala Public Jenkins --- M common/yarn-extras/pom.xml M ext-data-source/api/pom.xml M ext-data-source/sample/pom.xml M ext-data-source/test/pom.xml M fe/pom.xml M testdata/TableFlattener/pom.xml M testdata/pom.xml M tests/test-hive-udfs/pom.xml 8 files changed, 16 insertions(+), 16 deletions(-) Approvals: Philip Zeyliger: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/11352 ) Change subject: IMPALA-7508: Add Impala Python GDB module .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py File infra/gdb/impala-gdb.py: http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@1 PS1, Line 1: # I feel like this file should more properly be added to: $IMPALA_HOME/lib/python/impala_py_lib/gdb/ Perhaps Philz has an opinion on this? -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 23:06:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [diagnostics] Make --minidump dir consistent with Impala's --minidump path
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11353 ) Change subject: [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/528/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 Gerrit-Change-Number: 11353 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 23:06:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7445: enable missing buffer pool GC
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11350 ) Change subject: IMPALA-7445: enable missing buffer pool GC .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/11350/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11350/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-7445 is this the right JIRA? http://gerrit.cloudera.org:8080/#/c/11350/2//COMMIT_MSG@14 PS2, Line 14: new, soft, : mem_limit nit: commas? http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc File be/src/runtime/mem-tracker-test.cc: http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc@384 PS2, Line 384: Most of the GC attempts should fail nit: a bit confusing to say GC attempts fail since all GC attempts fail to reach the required mem. Maybe say that GC attempts get skipped due to backoff http://gerrit.cloudera.org:8080/#/c/11350/2/be/src/runtime/mem-tracker-test.cc@386 PS2, Line 386: EXPECT_FALSE(t.TryConsume(1, MemLimit::SOFT)); maybe add a 1ms sleep to ensure some on the gc attemps actually run -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 29 Aug 2018 23:05:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11352 ) Change subject: IMPALA-7508: Add Impala Python GDB module .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/527/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 22:49:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [diagnostics] Make --minidump dir consistent with Impala's --minidump path
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11353 ) Change subject: [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3094/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 Gerrit-Change-Number: 11353 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 22:42:35 + Gerrit-HasComments: No
[Impala-ASF-CR] [diagnostics] Make --minidump dir consistent with Impala's --minidump path
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11353 ) Change subject: [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 Gerrit-Change-Number: 11353 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 22:42:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 22:36:27 + Gerrit-HasComments: No
[Impala-ASF-CR] [diagnostics] Make --minidump dir consistent with Impala's --minidump path
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11353 ) Change subject: [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 Gerrit-Change-Number: 11353 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 22:36:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11352 ) Change subject: IMPALA-7508: Add Impala Python GDB module .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py File infra/gdb/impala-gdb.py: http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@29 PS1, Line 29: # Walk the threadlist to find fragment instance ids. Assumes IMPALA-6416, so Use triple " for docstring. http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@34 PS1, Line 34: def getFragmentInstances(self): Python's convention is to use snake_case instead of camelCase. http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@40 PS1, Line 40: while f is not None nit: can be simplified to while f: http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@80 PS1, Line 80: "Find IDs of all queries this impalad is currently executing." Use triple " for docstring. http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@95 PS1, Line 95: qid_hi + ':' + qid_low nit: usually it's preferable to use string format instead of + -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 22:29:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [diagnostics] Make --minidump dir consistent with Impala's --minidump path
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11353 Change subject: [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path .. [diagnostics] Make --minidump_dir consistent with Impala's --minidump_path Currently, the diagnostics script expects a full path to the actual directory to which process minidumps are written. This is however incosistent with Impala's configuration --minidump_path. Impala creates a subdirectory under FLAGS_minidump_path (for ex: /impalad) to which it writes the minidumps. This commit fixes the diagnostic script input --minidump_dir to be consistent with the above behavior from Impala. It now looks for minidumps under the directory <--minidump_path>/ The users of this script are expected to fix their input args accordingly. Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 --- M bin/diagnostics/collect_diagnostics.py 1 file changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/11353/1 -- To view, visit http://gerrit.cloudera.org:8080/11353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9e59f108a1f29a33768a39d0f4554d96e2dcd381 Gerrit-Change-Number: 11353 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11352 ) Change subject: IMPALA-7508: Add Impala Python GDB module .. Patch Set 1: (3 comments) Thanks for contributing this. Seems great to share this kind of thing, and also to have examples. I don't have much experience organizing these things, but the inheritance structure seems strange. I'd prefer not having inheritance and just composing your way into the common code. If you do have inheritance, you can avoid multiple-inheritance by having the superclass inherit from gdb.Command. http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py File infra/gdb/impala-gdb.py: http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@22 PS1, Line 22: import gdb Please add a README.md in this directory or include usage here. http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@28 PS1, Line 28: class ImpalaGDB: I think we usually use new-style classes. class ImpalaGDB(object): ... http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@62 PS1, Line 62: class FindFragmentInstances(gdb.Command, ImpalaGDB): : "Find all query fragment instance to thread Id mappings in this impalad." : : def __init__(self): : super(FindFragmentInstances, self).__init__( multiple-inheritance and inheritance in general seem unnecessary here. You could replace the class ImpalaGDB with: instances = get_fragment_instances(gdb) And then use that in your wrapper gdb Commands. -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 22:16:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add missing authorization in KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 6: Will merge the patch after more testing. -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 22:11:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11352 ) Change subject: IMPALA-7508: Add Impala Python GDB module .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py File infra/gdb/impala-gdb.py: http://gerrit.cloudera.org:8080/#/c/11352/1/infra/gdb/impala-gdb.py@96 PS1, Line 96: * flake8: E999 SyntaxError: invalid syntax -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 29 Aug 2018 22:08:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7508: Add Impala Python GDB module
Zoram Thanga has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11352 Change subject: IMPALA-7508: Add Impala Python GDB module .. IMPALA-7508: Add Impala Python GDB module This patch adds a new Impala Python GDB module, and implements a couple of covenience commands to make core dump analysis easier. The initial commands let us find queries and fragment instances currently executing in an impalad at the time the daemon crashes: (gdb) source impala-gdb.py (gdb) find-query-ids f74c863dff66a34d:1d983cc3 364525e12495932b:73f5dd02 bc4a3eec25481981:edda04b8 (gdb) find-fragment-instances Fragment Instance IdThread IDs 364525e12495932b:73f5dd0200a2 [69] 364525e12495932b:73f5dd020171 [196, 136] bc4a3eec25481981:edda04b801a8 [252, 237, 206] f74c863dff66a34d:1d983cc3009b [200, 14, 13, 12, 6, 5, 3, 2] f74c863dff66a34d:1d983cc3013a [4] The commands have only been tested with Impala 2.12, and are not expected to work with older versions since it uses ThreadDebugInfo stuff from IMPALA-6416. It is hoped that people contribute more commands to the module. Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a --- A infra/gdb/impala-gdb.py 1 file changed, 101 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11352/1 -- To view, visit http://gerrit.cloudera.org:8080/11352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I24e7026f2265954ed592d6f62110cf8cb2c2202a Gerrit-Change-Number: 11352 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. IMPALA-6644: Add last heartbeat timestamp into Statestore metric After this patch, the statestore keeps track of the time since the last heartbeat for each subscriber. It is exposed as a subscriber metric on the statestore debug page. It also adds a monitoring thread that periodically checks the last heartbeat timestamp for all subscribers and logs the IDs of those that have not been updated since the last periodic check. Testing: Added an end to end test to verify the 'sec_since_heartbeat' metric of a slow subscriber. Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Reviewed-on: http://gerrit.cloudera.org:8080/11052 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M tests/statestore/test_statestore.py M www/statestore_subscribers.tmpl 4 files changed, 104 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 16 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 15 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 22:05:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11344 ) Change subject: IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47a417a77df3f3030cf5f54fd2280b5e5e1fb77a Gerrit-Change-Number: 11344 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 21:46:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11344 ) Change subject: IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3093/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47a417a77df3f3030cf5f54fd2280b5e5e1fb77a Gerrit-Change-Number: 11344 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 21:46:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 29 Aug 2018 20:41:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Bump CDH_BUILD_NUMBER to 554247 This build includes the new HBase 2.1.0 snapshot as well as the new Kudu CDH tarballs, which a following patch will add functionality for downloading and building with. Testing: - Ran a full exhaustive build. Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Reviewed-on: http://gerrit.cloudera.org:8080/11349 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11344 ) Change subject: IMPALA-7502: ALTER TABLE RENAME should require ALL on the old table .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47a417a77df3f3030cf5f54fd2280b5e5e1fb77a Gerrit-Change-Number: 11344 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 20:25:51 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/526/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 20:04:12 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/525/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:57:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11280 ) Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/524/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 19:46:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7335: Fix a race in HdfsScanNode
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11337 ) Change subject: IMPALA-7335: Fix a race in HdfsScanNode .. Patch Set 3: > Patch Set 3: > > (1 comment) Modifying the SetDoneInternal() function to update the status sometimes causes a deadlock. This happens only for Parquet files. (I am still trying to figure out why) I think it makes sense to clean up the code rather than making the minor fix. -- To view, visit http://gerrit.cloudera.org:8080/11337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id470818846a5c69ad28a6ff695069702982aa793 Gerrit-Change-Number: 11337 Gerrit-PatchSet: 3 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 29 Aug 2018 19:34:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11280 ) Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/523/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 19:24:58 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Patch Set 1: Heh. I haven't been able to figure out how to hard-code that either. And I tried for a few hours. -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:20:11 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:19:30 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3092/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:18:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Patch Set 1: Yea, i wasn't going to start aggressively moving code over to Java 8 features. Mostly I was annoyed because when I load Impala in Eclipse, it always auto-selects Java7 JVM and the tests won't run until I go reconfigure the build path. -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:18:10 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11351 ) Change subject: Bump FE pom to Java 8 source/target version .. Patch Set 1: Code-Review+2 As I'm sure you're aware, this makes backporting to 2.x harder if anyone ever wants to, but I think it's a fair trade at this point. -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 19:15:50 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump FE pom to Java 8 source/target version
Hello Bharath Vissapragada, Philip Zeyliger, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/11351 to review the following change. Change subject: Bump FE pom to Java 8 source/target version .. Bump FE pom to Java 8 source/target version Our dependency on Hadoop 3 means we already required Java 8. This just fixes the pom so we are compiling to Java 8 classes. This will also allow us to start using some Java 8 features like lambdas for more concise code. Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 --- M common/yarn-extras/pom.xml M ext-data-source/api/pom.xml M ext-data-source/sample/pom.xml M ext-data-source/test/pom.xml M fe/pom.xml M testdata/TableFlattener/pom.xml M testdata/pom.xml M tests/test-hive-udfs/pom.xml 8 files changed, 16 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/11351/1 -- To view, visit http://gerrit.cloudera.org:8080/11351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0a5e4cf3f4171eecf218f6d4dd7cdfece9dc9152 Gerrit-Change-Number: 11351 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] Add missing authorization in KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 6: Carry Sailesh's +1 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:09:11 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/11331 )
Change subject: Add missing authorization in KRPC
..
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/authentication.cc
File be/src/rpc/authentication.cc:
http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/authentication.cc@734
PS5, Line 734: return Status(Substitute("Bad --krb5_ccname value: $0 is not
an absolute file path",
> nit: use the user-facing name --krb5_ccname instead of FLAGS_... which is i
Done
http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/rpc-mgr-kerberized-test.cc
File be/src/rpc/rpc-mgr-kerberized-test.cc:
http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/rpc-mgr-kerberized-test.cc@114
PS5, Line 114: controller.Reset();
: rpc_status =
: FromKuduStatus(scan_proxy
> So, this is passing only because the Authorize() function for ScanMemServic
Yes, I have some comments about it on line 99-100 but yes, I should move them
to a location closer to the actual execution of the test to make it clearer.
http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/rpc-mgr.cc@172
PS5, Line 172: LOG(ERROR) << Substitute("Rejecting unauthorized access to
$0 from $1. Expected "
> sorry, think I wasn't entirely clear. I think the error message should stil
Lol. It's indeed a bit scary to not have the word "Reject" in the error
message. Fixed. Thanks.
--
To view, visit http://gerrit.cloudera.org:8080/11331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5
Gerrit-Change-Number: 11331
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Ho
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 29 Aug 2018 19:08:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Add missing authorization in KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 19:09:17 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Hello Sailesh Mukil, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11331 to look at the new patch set (#6). Change subject: Add missing authorization in KRPC .. Add missing authorization in KRPC In 2.12.0, Impala adopted Kudu RPC library for certain backened services (TransmitData(), EndDataStream()). While the implementation uses Kerberos for authenticating users connecting to the backend services, there is no authorization implemented. This is a regression from the Thrift based implementation because it registered a SASL callback (SaslAuthorizeInternal) to be invoked during the connection negotiation. With this regression, an unauthorized but authenticated user may invoke RPC calls to Impala backend services. This change fixes the issue above by overriding the default authorization method for the DataStreamService. The authorization method will only let authenticated principal which matches FLAGS_principal / FLAGS_be_principal to access the service. Also added a new startup flag --krb5_ccname to allow users to customize the locations of the Kerberos credentials cache. Testing done: 1. Added a new test case in rpc-mgr-kerberized-test.cc to confirm an unauthorized user is not allowed to access the service. 2. Ran some queries in a Kerberos enabled cluster to make sure there is no error. 3. Exhaustive builds. Thanks to Todd Lipcon for pointing out the problem and his guidance on the fix. Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 --- M be/src/common/global-flags.cc M be/src/rpc/authentication.cc M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc R be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/thrift-server-test.cc M be/src/runtime/data-stream-test.cc M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/testutil/mini-kdc-wrapper.cc M be/src/testutil/mini-kdc-wrapper.h M be/src/util/auth-util.cc M be/src/util/auth-util.h M bin/rat_exclude_files.txt M common/protobuf/data_stream_service.proto A common/protobuf/kudu M common/protobuf/rpc_test.proto 19 files changed, 330 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11331/6 -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Hello Bharath Vissapragada, Tianyi Wang, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11280 to look at the new patch set (#5). Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. IMPALA-7469. Invalidate LocalCatalog cache based on topic updates This implements cache invalidation inside CatalogdMetaProvider. The design is as follows: - when the catalogd collects updates into the statestore topic, it now adds an additional entry for each table and database. These additional entries are minimal - they only include the object's name, but no metadata. This new behavior is conditional on a new flag --catalog_topic_mode. The default mode is to keep the old style, but it can be configured to mixed (support both v1 and v2) or v2-only. - the old-style topic entries are prefixed with a '1:' whereas the new minimal entries are prefixed with a '2:'. The impalad will subscribe to one or the other prefix depending on whether it is running with --use_local_catalog. Thus, old impalads will not be confused by the new entries and vice versa. - when the impalad gets these topic updates, it forwards them through to the catalog implementation. The LocalCatalog implementation forwards them to the CatalogdMetaProvider, which uses them to invalidate cached metadata as appropriate. This patch includes some basic unit tests. I also did some manual testing by connecting to different impalads and verifying that a session connected to impalad #1 saw the effects of DDLs made by impalad #2 within a short period of time (the statestore topic update frequency). Existing end-to-end tests cover these code paths pretty thoroughly: - if we didn't automatically invalidate the cache on a coordinator in response to DDL operations, then any test which expects to "read its own writes" (eg access a table after creating one) would fail - if we didn't propagate invalidations via the statestore, then all of the tests that use sync_ddl would fail. I verified the test coverage above using some of the tests in test_ddl.py -- I selectively commented out a few of the invalidation code paths in the new code and verified that tests failed until I re-introduced them. Along the way I also improved test_ddl so that, when this code is broken, it properly fails with a timeout. It also has a bit of expanded coverage for both the SYNC_DDL and non-SYNC cases. I also wrote a new custom-cluster test for LocalCatalog that verifies a few of the specific edge cases like detecting catalogd restart. One notable exception here is the implementation of INVALIDATE METADATA This turned out to be complex to implement, so I left a lengthy TODO describing the issue and filed a JIRA. Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 --- M be/src/catalog/catalog-server.cc M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/common/Pair.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_local_catalog.py M tests/metadata/test_ddl.py 21 files changed, 776 insertions(+), 103 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/11280/5 -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 15 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 18:55:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3091/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 15 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 18:55:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 14: Code-Review+2 Carrying Todd's +2 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 14 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 18:54:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11280 ) Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/11280/4/tests/custom_cluster/test_local_catalog.py File tests/custom_cluster/test_local_catalog.py: http://gerrit.cloudera.org:8080/#/c/11280/4/tests/custom_cluster/test_local_catalog.py@25 PS4, Line 25: class TestCompactCatalogUpdates(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/11280/4/tests/custom_cluster/test_local_catalog.py@49 PS4, Line 49: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/11280/4/tests/custom_cluster/test_local_catalog.py@66 PS4, Line 66: @ flake8: E303 too many blank lines (2) -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 18:46:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 14 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 18:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11280 ) Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. Patch Set 3: (4 comments) Rebased, addressed the lint checks, and also fixed up the partial-catalog test so that it doesn't run into the new check for the catalogd configuration http://gerrit.cloudera.org:8080/#/c/11280/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/11280/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@308 PS3, Line 308: checkResponse(resp.catalog_info != null && resp.catalog_info.db_names != null, req, > line too long (95 > 90) Done http://gerrit.cloudera.org:8080/#/c/11280/3/tests/custom_cluster/test_local_catalog.py File tests/custom_cluster/test_local_catalog.py: http://gerrit.cloudera.org:8080/#/c/11280/3/tests/custom_cluster/test_local_catalog.py@25 PS3, Line 25: class TestCompactCatalogUpdates(CustomClusterTestSuite): > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/11280/3/tests/custom_cluster/test_local_catalog.py@49 PS3, Line 49: > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/11280/3/tests/custom_cluster/test_local_catalog.py@66 PS3, Line 66: @ > flake8: E303 too many blank lines (2) Done -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 29 Aug 2018 18:45:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
Hello Bharath Vissapragada, Tianyi Wang, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11280 to look at the new patch set (#4). Change subject: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates .. IMPALA-7469. Invalidate LocalCatalog cache based on topic updates This implements cache invalidation inside CatalogdMetaProvider. The design is as follows: - when the catalogd collects updates into the statestore topic, it now adds an additional entry for each table and database. These additional entries are minimal - they only include the object's name, but no metadata. This new behavior is conditional on a new flag --catalog_topic_mode. The default mode is to keep the old style, but it can be configured to mixed (support both v1 and v2) or v2-only. - the old-style topic entries are prefixed with a '1:' whereas the new minimal entries are prefixed with a '2:'. The impalad will subscribe to one or the other prefix depending on whether it is running with --use_local_catalog. Thus, old impalads will not be confused by the new entries and vice versa. - when the impalad gets these topic updates, it forwards them through to the catalog implementation. The LocalCatalog implementation forwards them to the CatalogdMetaProvider, which uses them to invalidate cached metadata as appropriate. This patch includes some basic unit tests. I also did some manual testing by connecting to different impalads and verifying that a session connected to impalad #1 saw the effects of DDLs made by impalad #2 within a short period of time (the statestore topic update frequency). Existing end-to-end tests cover these code paths pretty thoroughly: - if we didn't automatically invalidate the cache on a coordinator in response to DDL operations, then any test which expects to "read its own writes" (eg access a table after creating one) would fail - if we didn't propagate invalidations via the statestore, then all of the tests that use sync_ddl would fail. I verified the test coverage above using some of the tests in test_ddl.py -- I selectively commented out a few of the invalidation code paths in the new code and verified that tests failed until I re-introduced them. Along the way I also improved test_ddl so that, when this code is broken, it properly fails with a timeout. It also has a bit of expanded coverage for both the SYNC_DDL and non-SYNC cases. I also wrote a new custom-cluster test for LocalCatalog that verifies a few of the specific edge cases like detecting catalogd restart. One notable exception here is the implementation of INVALIDATE METADATA This turned out to be complex to implement, so I left a lengthy TODO describing the issue and filed a JIRA. Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 --- M be/src/catalog/catalog-server.cc M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/common/Pair.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_local_catalog.py M tests/metadata/test_ddl.py 21 files changed, 776 insertions(+), 103 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/11280/4 -- To view, visit http://gerrit.cloudera.org:8080/11280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984 Gerrit-Change-Number: 11280 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] Add bin/jenkins/finalize.sh
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/11322 ) Change subject: Add bin/jenkins/finalize.sh .. Patch Set 2: Code-Review+1 This looks reasonable to me. +1 for now in case others have comments as well. -- To view, visit http://gerrit.cloudera.org:8080/11322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4d223b6e441f01a5edad9d6f9ca69c6de023461 Gerrit-Change-Number: 11322 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Wed, 29 Aug 2018 18:43:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7445: enable missing buffer pool GC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11350 ) Change subject: IMPALA-7445: enable missing buffer pool GC .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/521/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 29 Aug 2018 18:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. IMPALA-6644: Add last heartbeat timestamp into Statestore metric After this patch, the statestore keeps track of the time since the last heartbeat for each subscriber. It is exposed as a subscriber metric on the statestore debug page. It also adds a monitoring thread that periodically checks the last heartbeat timestamp for all subscribers and logs the IDs of those that have not been updated since the last periodic check. Testing: Added an end to end test to verify the 'sec_since_heartbeat' metric of a slow subscriber. Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M tests/statestore/test_statestore.py M www/statestore_subscribers.tmpl 4 files changed, 104 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11052/14 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 14 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7445: enable missing buffer pool GC
Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11350 ) Change subject: IMPALA-7445: enable missing buffer pool GC .. IMPALA-7445: enable missing buffer pool GC This was not hooked up as an oversight, and means that sometimes queries can spill or OOM because of accumulated free buffers and pages in the buffer pool. This patch ensures that we try to free up unused memory from the buffer pool before we start failing memory allocations. I was somewhat concerned about an interaction between the new, soft, mem_limit and the GC. If the system is near or over the soft process memory limit, it's possible that many spilling operators or HDFS scans could be repeatedly bumping up against the soft memory limit and doing failed GCs. With hard memory limits, failed GCs lead to many queries failing and the situation resolving itself, but with soft memory limits the queries will likely be able to continue running while doing many GCs. My solution was to back off soft GCs for a 100ms period after a failure. Testing: Added unit tests. Looped the query from IMPALA-7442 to confirm it didn't fail. Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 --- M be/src/common/init.cc M be/src/runtime/exec-env.cc M be/src/runtime/mem-tracker-test.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h 5 files changed, 140 insertions(+), 75 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/11350/2 -- To view, visit http://gerrit.cloudera.org:8080/11350 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10 Gerrit-Change-Number: 11350 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 17:32:15 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3090/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 29 Aug 2018 17:15:06 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 29 Aug 2018 17:15:05 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Patch Set 2: Code-Review+2 (1 comment) carrying forward http://gerrit.cloudera.org:8080/#/c/11349/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11349/1//COMMIT_MSG@11 PS1, Line 11: downloading and building with. > building Done -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 29 Aug 2018 17:14:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Hello Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11349 to look at the new patch set (#2). Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Bump CDH_BUILD_NUMBER to 554247 This build includes the new HBase 2.1.0 snapshot as well as the new Kudu CDH tarballs, which a following patch will add functionality for downloading and building with. Testing: - Ran a full exhaustive build. Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/11349/2 -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/520/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 17:11:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 13: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/519/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 16:59:42 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing authorization in KRPC
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/11331 ) Change subject: Add missing authorization in KRPC .. Patch Set 5: Code-Review+1 (1 comment) Testing side LGTM. http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/rpc-mgr-kerberized-test.cc File be/src/rpc/rpc-mgr-kerberized-test.cc: http://gerrit.cloudera.org:8080/#/c/11331/5/be/src/rpc/rpc-mgr-kerberized-test.cc@114 PS5, Line 114: rpc_status = : FromKuduStatus(scan_proxy->ScanMem(scan_request, &scan_response, &controller)); : EXPECT_TRUE(rpc_status.ok()); So, this is passing only because the Authorize() function for ScanMemServiceImpl always returns true right? I had to read across files to figure that out. I think adding a comment like the following should make the test easier to understand: "ScanMemServiceImpl authorizes everyone connecting to the service, so we expect it to pass even though we have a different principal name". And the opposite for PingServiceImpl. -- To view, visit http://gerrit.cloudera.org:8080/11331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f82dee5e721f2ed23e75fd91abbc6ab7addd4c5 Gerrit-Change-Number: 11331 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 16:55:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11349 ) Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/11349/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11349/1//COMMIT_MSG@11 PS1, Line 11: downloading and biulding with. building -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 29 Aug 2018 16:38:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11297 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/518/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc87a84c34935d0d5841c7f5528eb802527fa809 Gerrit-Change-Number: 11297 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 29 Aug 2018 16:36:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 554247
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11349 Change subject: Bump CDH_BUILD_NUMBER to 554247 .. Bump CDH_BUILD_NUMBER to 554247 This build includes the new HBase 2.1.0 snapshot as well as the new Kudu CDH tarballs, which a following patch will add functionality for downloading and biulding with. Testing: - Ran a full exhaustive build. Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/11349/1 -- To view, visit http://gerrit.cloudera.org:8080/11349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ieae9581997367895be5f1f8c730968d375182d94 Gerrit-Change-Number: 11349 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 13: (1 comment) > Patch Set 12: > > (1 comment) http://gerrit.cloudera.org:8080/#/c/11052/12/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/11052/12/be/src/statestore/statestore.cc@78 PS12, Line 78: 6000 > this went from 1 minute to 6 seconds here, was that intentional? It was a mistake. Fixed it. -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 29 Aug 2018 16:15:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. IMPALA-6644: Add last heartbeat timestamp into Statestore metric After this patch, the statestore keeps track of the time since the last heartbeat for each subscriber. It is exposed as a subscriber metric on the statestore debug page. It also adds a monitoring thread that periodically checks the last heartbeat timestamp for all subscribers and logs the IDs of those that have not been updated since the last periodic check. Testing: Added an end to end test to verify the 'sec_since_heartbeat' metric of a slow subscriber. Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M tests/statestore/test_statestore.py M www/statestore_subscribers.tmpl 4 files changed, 104 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11052/13 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11297 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 8: Code-Review+1 carry -- To view, visit http://gerrit.cloudera.org:8080/11297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc87a84c34935d0d5841c7f5528eb802527fa809 Gerrit-Change-Number: 11297 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 29 Aug 2018 15:52:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11297 ) Change subject: IMPALA-7477: Batch-oriented query set construction .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/11297/7/be/src/service/hs2-util.cc File be/src/service/hs2-util.cc: http://gerrit.cloudera.org:8080/#/c/11297/7/be/src/service/hs2-util.cc@137 PS7, Line 137: values > I think it should be 'nulls'. Thanks for catching -- To view, visit http://gerrit.cloudera.org:8080/11297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc87a84c34935d0d5841c7f5528eb802527fa809 Gerrit-Change-Number: 11297 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 29 Aug 2018 15:52:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7477: Batch-oriented query set construction
Hello Zoltan Borok-Nagy, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11297 to look at the new patch set (#8). Change subject: IMPALA-7477: Batch-oriented query set construction .. IMPALA-7477: Batch-oriented query set construction Rework the row-by-row construction of query result sets in PlanRootSink so that it materialises an output column at a time. Make some minor optimisations like preallocating output vectors and initialising strings more efficiently. My intent is both to make this faster and to make the QueryResultSet interface better before IMPALA-4268 does a bunch of surgery on this part of the code. Testing: Ran core tests. Perf: Downloaded tpch_parquet.orders via JDBC driver. Before: 3.01s, After: 2.57s. Downloaded l_orderkey from tpch_parquet.lineitem. Before: 1.21s, After: 1.08s. Change-Id: Ibc87a84c34935d0d5841c7f5528eb802527fa809 --- M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/hs2-util.cc M be/src/service/hs2-util.h M be/src/service/query-result-set.cc M be/src/service/query-result-set.h 6 files changed, 320 insertions(+), 161 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/11297/8 -- To view, visit http://gerrit.cloudera.org:8080/11297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibc87a84c34935d0d5841c7f5528eb802527fa809 Gerrit-Change-Number: 11297 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
