[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 7: (11 comments) http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@25 PS7, Line 25: d flake8: E501 line too long (162 > 90 characters) http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@29 PS7, Line 29: from thrift.protocol import TCompactProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@30 PS7, Line 30: from thrift.TSerialization import deserialize flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@31 PS7, Line 31: from RuntimeProfile.ttypes import TRuntimeProfileTree flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@33 PS7, Line 33: import argparse flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@34 PS7, Line 34: import base64 flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@35 PS7, Line 35: import datetime flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@36 PS7, Line 36: import sys flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@37 PS7, Line 37: import zlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@39 PS7, Line 39: import matplotlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/7/bin/plot_profile_resource_usage.py@41 PS7, Line 41: import matplotlib.pyplot as plt flake8: E402 module level import not at top of file -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Dec 2018 07:32:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Hello Michael Ho, Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12069 to look at the new patch set (#7). Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. Example: https://user-images.githubusercontent.com/151514/49830685-bb7efd80-fd46-11e8-8e23-9f5bc47635c1.png This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 883 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/7 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Hello Michael Ho, Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12069 to look at the new patch set (#6). Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. Example: https://user-images.githubusercontent.com/151514/49830685-bb7efd80-fd46-11e8-8e23-9f5bc47635c1.png This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 884 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/6 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7844: HAVING clause cannot support ordinals
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/11955 )
Change subject: IMPALA-7844: HAVING clause cannot support ordinals
..
Patch Set 4:
(9 comments)
Minor comments. Core of the patch lgtm. Zoltan, could you take a pass too? I
see that you made some significant refactoring here in IMPALA-5191.
http://gerrit.cloudera.org:8080/#/c/11955/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11955/4//COMMIT_MSG@21
PS4, Line 21: opeators
typo
http://gerrit.cloudera.org:8080/#/c/11955/4//COMMIT_MSG@21
PS4, Line 21: than
then
http://gerrit.cloudera.org:8080/#/c/11955/4//COMMIT_MSG@24
PS4, Line 24: unusal
typo
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
File fe/src/main/java/org/apache/impala/analysis/QueryStmt.java:
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@329
PS4, Line 329: resoluton
typo
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@336
PS4, Line 336: Substitutes
Not sure if the substitution is happening here. As I understand it, we just
resolve the ordinal/alias and return an analyzed expr clone which the callers
can substitute anywhere.
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@353
PS4, Line 353: allowOrdinal
mention this in the method doc
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java@389
PS4, Line 389: Analyze it so all expressions exit
: // this method analyzed.
Isn't this happening L372?
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1074
PS4, Line 1074: "if(true, 7, int_col)");
Test -ve ordinal values?
http://gerrit.cloudera.org:8080/#/c/11955/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1124
PS4, Line 1124: @Ignore("IMPALA-7844: Ordinals not supported in HAVING")
Don't think this is the right way. Instead, convert them into proper
AnalysisError() calls?
--
To view, visit http://gerrit.cloudera.org:8080/11955
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b9f9e8c60fe2b25e20c57c2ffc31d8e59d5861
Gerrit-Change-Number: 11955
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Wed, 12 Dec 2018 06:15:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7889: Write new logical types in Parquet
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12004 )
Change subject: IMPALA-7889: Write new logical types in Parquet
..
Patch Set 7:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/12004/7/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:
http://gerrit.cloudera.org:8080/#/c/12004/7/be/src/exec/parquet/parquet-metadata-utils.cc@158
PS7, Line 158: static bool IsEncodingSupported(parquet::Encoding::type e) {
There's already a "IsSupportedType()" in the anonymous namespace above, I think
we can move this there, and add a comment to explain what it does.
http://gerrit.cloudera.org:8080/#/c/12004/7/be/src/exec/parquet/parquet-metadata-utils.cc@290
PS7, Line 290: namespace {
I'd consider moving all anonymous helpers up into one anonymous namespace
http://gerrit.cloudera.org:8080/#/c/12004/5/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/12004/5/tests/query_test/test_insert_parquet.py@388
PS5, Line 388: found = False
> Sorry, I forgot this one in patch set 6.
You could rename _find_schema to _get_schema() if you feel that that would
express more clearly that it actually has to exist, and then add an assert
there instead of here (since there doesn't seem to be a case where it doesn't
exist). I don't feel strongly about this.
http://gerrit.cloudera.org:8080/#/c/12004/7/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/12004/7/tests/query_test/test_insert_parquet.py@37
PS7, Line 37: from tests.util.get_parquet_metadata import decode_stats_value, \
nit: Wrap these in parentheses, and while you're here the ones above, too. (see
https://www.python.org/dev/peps/pep-0328/)
http://gerrit.cloudera.org:8080/#/c/12004/7/tests/query_test/test_insert_parquet.py@388
PS7, Line 388: found = False
I think you can shorten this to something like:
keys = [k for k, v in obj_dict.iteritems() if v is not None]
assert keys == [var_name]
Or make it one line if you prefer
http://gerrit.cloudera.org:8080/#/c/12004/7/tests/query_test/test_insert_parquet.py@410
PS7, Line 410: 8: ConvertedType.INT_8,
nit: I think we indent 4 spaces here
http://gerrit.cloudera.org:8080/#/c/12004/7/tests/query_test/test_insert_parquet.py@456
PS7, Line 456: # This test will break once INT64 becomes the default
Parquet type for TIMESTAMP
Is there actually a Jira we can add here? If not that's ok, too
--
To view, visit http://gerrit.cloudera.org:8080/12004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f377950845683ab9c6dea79f4c54db0359d0b91
Gerrit-Change-Number: 12004
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Comment-Date: Wed, 12 Dec 2018 05:31:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3554/ -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Dec 2018 05:11:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12073 ) Change subject: IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" .. IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" The fix for IMPALA-5929 introduced a bug that produced wrong results. This bug is detailed in IMPALA-7960. Reverting for now. This reverts commit 545163bb0a5c86aa02652d0557871f5b694a6c82. Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Reviewed-on: http://gerrit.cloudera.org:8080/12073 Reviewed-by: Zoram Thanga Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M testdata/workloads/functional-query/queries/QueryTest/exprs.test 3 files changed, 0 insertions(+), 100 deletions(-) Approvals: Zoram Thanga: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12073 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Gerrit-Change-Number: 12073 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12073 ) Change subject: IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12073 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Gerrit-Change-Number: 12073 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Wed, 12 Dec 2018 04:47:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5474: Adding a trivial subquery turns error into warning
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12022 ) Change subject: IMPALA-5474: Adding a trivial subquery turns error into warning .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e Gerrit-Change-Number: 12022 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Wed, 12 Dec 2018 04:32:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. Patch Set 6: (3 comments) Some small readability nits but otherwise lgtm http://gerrit.cloudera.org:8080/#/c/12047/6/tests/shell/util.py File tests/shell/util.py: http://gerrit.cloudera.org:8080/#/c/12047/6/tests/shell/util.py@97 PS6, Line 97: False if not expect_success : else wait_until_connected This looks equivalent to "expect_success and wait_until_connected". I think it might be more readable to define a new variable above, and it'll likely use the same number of lines. http://gerrit.cloudera.org:8080/#/c/12047/6/tests/shell/util.py@138 PS6, Line 138: args is None or "--quiet" not in args) You can simplify this to args and "--quiet" not in args The difference between "not args" and "args is None" doesn't matter here. http://gerrit.cloudera.org:8080/#/c/12047/6/tests/shell/util.py@141 PS6, Line 141: while time.time() - start_time < timeout: You could also do: while ... and not connected: and remove the if: break, but I'm fine with either way. -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 12 Dec 2018 04:35:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12068 ) Change subject: IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1585/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 Gerrit-Change-Number: 12068 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 12 Dec 2018 02:12:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7948: part 1: initial docker container build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12074 ) Change subject: IMPALA-7948: part 1: initial docker container build .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e Gerrit-Change-Number: 12074 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 12 Dec 2018 01:57:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7948: part 1: initial docker container build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12074 ) Change subject: IMPALA-7948: part 1: initial docker container build .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1584/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e Gerrit-Change-Number: 12074 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 12 Dec 2018 01:54:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr.
Andrew Sherman has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12068 ) Change subject: IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. .. IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. These two classes evaluate scalar expressions. Previously codegen was done by calling ScalarExpr::GetCodegendComputeFnWrapper which generates a static method that calls the scalar expression evaluation methods. Make this more efficient by using cross-compiled code which is customized at codegen time. Add cross-compiled files is-not-empty-predicate-ir.cc and valid-tuple-id-ir.cc These files contain the methods that will be specialized during code generation These methods have to be static to match the expected method signatures Refactor the existing scalar methods so that they call a new internal static method which contains the implementation of the scalar expression This static method is called from both the old member function and the new cross-compiled method. IMPALA-7657 also requests replacing GetCodegendComputeFnWrapper() in TupleIsNullPredicate. In the current Impala code this method is never called. This is because TupleIsNullPredicate is always wrapped in an IfExpr. This is always codegen'd by IfExpr's GetCodegendComputeFnWrapper() method. There is a separate Jira IMPALA-7655 to improve codegen of IfExpr. Minor corrections: Correct the link to llvm tutorial in LlvmCodegen. Make a method private in TupleIsNullPredicate.java. TESTING: The changed scalar expressions are well exercised by current tests. Ran end-to-end tests. Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.h M be/src/exprs/CMakeLists.txt A be/src/exprs/is-not-empty-predicate-ir.cc M be/src/exprs/is-not-empty-predicate.cc M be/src/exprs/is-not-empty-predicate.h M be/src/exprs/slot-ref.cc A be/src/exprs/valid-tuple-id-ir.cc M be/src/exprs/valid-tuple-id.cc M be/src/exprs/valid-tuple-id.h M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java 12 files changed, 206 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/12068/3 -- To view, visit http://gerrit.cloudera.org:8080/12068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 Gerrit-Change-Number: 12068 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-5474: Adding a trivial subquery turns error into warning
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/12022 )
Change subject: IMPALA-5474: Adding a trivial subquery turns error into warning
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12022/3/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/12022/3/shell/impala_client.py@531
PS3, Line 531: type_str = "WARNINGS" if warn is True else "ERROR"
The problem with this approach (and the reason I suggested the approach of
calling fetch()) is that the log can also include warnings as well as the error
that ultimately caused the query failure. E.g. I constructed this query that
has both an analysis warning and a runtime warning:
[localhost:21000] default> set abort_on_error=1; select id, cnt from
functional_parquet.bad_column_metadata t cross join /*+badhint*/ (select
base64decode('foo') cnt) u;
ABORT_ON_ERROR set to 1
Query: select id, cnt from functional_parquet.bad_column_metadata t cross
join /*+badhint*/ (select base64decode('foo') cnt) u
Query submitted at: 2018-12-11 17:39:41 (Coordinator:
http://tarmstrong-box:25000)
Query progress can be monitored at:
http://tarmstrong-box:25000/query_plan?query_id=bb407635ac88a3e2:875df25d
ERROR: Column metadata states there are 11 values, but read 10 values from
column id.
file=hdfs://localhost:20500/test-warehouse/bad_column_metadata_parquet/bad_column_metadata.parquet
JOIN hint not recognized: badhint
UDF WARNING: Invalid base64 string; input length is 3, which is not a
multiple of 4.
If we go down the other code path where the fetch() call fails, we only get the
query status, which does not include previous warnings, which is a little weird.
This could also get confusing with the errors and warnings mixed but it sort-of
works out because the query status is always the first thing in the log:
https://github.com/apache/impala/blob/91c754bf0ac707d0b2cee23c11ef41d7c81dd0f0/be/src/service/impala-beeswax-server.cc#L278
Anyway.. this is still an improvement and probably ok just so long as we
document it in the code - can you add a comment to get_warn_or_error_log that
explains what the log includes?
--
To view, visit http://gerrit.cloudera.org:8080/12022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e
Gerrit-Change-Number: 12022
Gerrit-PatchSet: 3
Gerrit-Owner: Yongjun Zhang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongjun Zhang
Gerrit-Comment-Date: Wed, 12 Dec 2018 01:48:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7948: part 1: initial docker container build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12074 ) Change subject: IMPALA-7948: part 1: initial docker container build .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/12074/3/docker/setup_build_context.sh File docker/setup_build_context.sh: http://gerrit.cloudera.org:8080/#/c/12074/3/docker/setup_build_context.sh@87 PS3, Line 87: ln -s "$IMPALA_HOME"/fe/target/dependency/*.jar "$IMPALA_HOME"/fe/target/impala-frontend-*.jar \ line too long (96 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e Gerrit-Change-Number: 12074 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Dec 2018 01:20:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7948: part 1: initial docker container build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12074 ) Change subject: IMPALA-7948: part 1: initial docker container build .. Patch Set 4: (1 comment) This is still missing a lot of stuff, but also doesn't break anything existing and adds a lot of the infrastructure, so I was hoping to get it in while I build more things on top of it. http://gerrit.cloudera.org:8080/#/c/12074/3/docker/setup_build_context.sh File docker/setup_build_context.sh: http://gerrit.cloudera.org:8080/#/c/12074/3/docker/setup_build_context.sh@87 PS3, Line 87: ln -s "$IMPALA_HOME"/fe/target/dependency/*.jar \ > line too long (96 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/12074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e Gerrit-Change-Number: 12074 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 12 Dec 2018 01:25:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7948: part 1: initial docker container build
Tim Armstrong has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12074 ) Change subject: IMPALA-7948: part 1: initial docker container build .. IMPALA-7948: part 1: initial docker container build This builds an impala_base container that has all of the build artifacts required to run the impala processes, then builds impalad, catalogd and statestore containers based on that with the right ports exposed. The images are based on the Ubuntu 16.04 image to align with the most common development environment. The container build process is integrated with CMake and is designed to integrate with the rest of the build so that the container build depends on the artifacts that will go into the container. You can build the images with the following command, which will create images called "impala_base", "impalad", "catalogd" and "statestored": ninja -j $IMPALA_BUILD_THREADS docker_images The images need some refinement to be truly useful. The following will be done in future patches: * IMPALA-7947 - integrate with start-impala-cluster.py to automatically create docker network with containers running on it * Mechanism to pass in command-line flags * Mechanisms to update the various config files to point to the docker host rather than "localhost", which doesn't point to the right thing inside the container. * Mechanisms to set mem_limit, JVM heap sizes, etc, automatically. * Mapping /etc/localtime from host Testing: Manually started up the containers connected to a user-defined bridge network, tweaked the configurations to point to the HMS/HDFS/etc running on my host. I then used "docker ps" to figure out the port mappings for beeswax and debug webserver. Confirmed that I could run a query and access debug pages: $ impala-shell.sh -i localhost:32860 -q "select coordinator()" Starting Impala Shell without Kerberos authentication Opened TCP connection to localhost:32860 Connected to localhost:32860 Server version: impalad version 3.1.0-SNAPSHOT DEBUG (build d7870fe03645490f95bd5ffd4a2177f90eb2f3c0) Query: select coordinator() Query submitted at: 2018-12-11 15:51:04 (Coordinator: http://8063e77ce999:25000) Query progress can be monitored at: http://8063e77ce999:25000/query_plan?query_id=1b4d03f0f0f1fcfb:b0b37e50 +---+ | coordinator() | +---+ | 8063e77ce999 | +---+ Fetched 1 row(s) in 0.11s Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e --- M .gitignore M CMakeLists.txt M be/src/service/CMakeLists.txt M bin/start-catalogd.sh M bin/start-statestored.sh A docker/CMakeLists.txt M docker/README.md A docker/catalogd/Dockerfile A docker/impala_base/Dockerfile A docker/impalad/Dockerfile A docker/run_with_classpath.sh A docker/setup_build_context.sh A docker/statestored/Dockerfile M fe/CMakeLists.txt 14 files changed, 391 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/12074/4 -- To view, visit http://gerrit.cloudera.org:8080/12074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e Gerrit-Change-Number: 12074 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7948: part 1: initial docker container build
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12074 Change subject: IMPALA-7948: part 1: initial docker container build .. IMPALA-7948: part 1: initial docker container build This builds an impala_base container that has all of the build artifacts required to run the impala processes, then builds impalad, catalogd and statestore containers based on that with the right ports exposed. The images are based on the Ubuntu 16.04 image to align with the most common development environment. The container build process is integrated with CMake and is designed to integrate with the rest of the build so that the container build depends on the artifacts that will go into the container. You can build the images with the following command, which will create images called "impala_base", "impalad", "catalogd" and "statestored": ninja -j $IMPALA_BUILD_THREADS docker_images The images need some refinement to be truly useful. The following will be done in future patches: * IMPALA-7947 - integrate with start-impala-cluster.py to automatically create docker network with containers running on it * Mechanism to pass in command-line flags * Mechanisms to update the various config files to point to the docker host rather than "localhost", which doesn't point to the right thing inside the container. * Mechanisms to set mem_limit, JVM heap sizes, etc, automatically. * Mapping /etc/localtime from host Testing: Manually started up the containers connected to a user-defined bridge network, tweaked the configurations to point to the HMS/HDFS/etc running on my host. I then used "docker ps" to figure out the port mappings for beeswax and debug webserver. Confirmed that I could run a query and access debug pages: $ impala-shell.sh -i localhost:32860 -q "select coordinator()" Starting Impala Shell without Kerberos authentication Opened TCP connection to localhost:32860 Connected to localhost:32860 Server version: impalad version 3.1.0-SNAPSHOT DEBUG (build d7870fe03645490f95bd5ffd4a2177f90eb2f3c0) Query: select coordinator() Query submitted at: 2018-12-11 15:51:04 (Coordinator: http://8063e77ce999:25000) Query progress can be monitored at: http://8063e77ce999:25000/query_plan?query_id=1b4d03f0f0f1fcfb:b0b37e50 +---+ | coordinator() | +---+ | 8063e77ce999 | +---+ Fetched 1 row(s) in 0.11s Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e --- M .gitignore M CMakeLists.txt M be/src/service/CMakeLists.txt M bin/start-catalogd.sh M bin/start-statestored.sh A docker/CMakeLists.txt M docker/README.md A docker/catalogd/Dockerfile A docker/impala_base/Dockerfile A docker/impalad/Dockerfile A docker/run_with_classpath.sh A docker/setup_build_context.sh A docker/statestored/Dockerfile M fe/CMakeLists.txt 14 files changed, 390 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/12074/3 -- To view, visit http://gerrit.cloudera.org:8080/12074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifea707aa3cc23e4facda8ac374160c6de23ffc4e Gerrit-Change-Number: 12074 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12073 ) Change subject: IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1582/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12073 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Gerrit-Change-Number: 12073 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Wed, 12 Dec 2018 01:19:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5474: Adding a trivial subquery turns error into warning
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12022 ) Change subject: IMPALA-5474: Adding a trivial subquery turns error into warning .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1581/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e Gerrit-Change-Number: 12022 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Wed, 12 Dec 2018 01:03:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12068 ) Change subject: IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1580/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 Gerrit-Change-Number: 12068 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 12 Dec 2018 00:52:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3554/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Dec 2018 00:54:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5474: Adding a trivial subquery turns error into warning
Yongjun Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/12022 ) Change subject: IMPALA-5474: Adding a trivial subquery turns error into warning .. Patch Set 3: > Yeah WARNING isn't a query state, it's just that the error log > mechanism can be used to return warnings that don't fail the query. > > It should be valid to call fetch on any query. Since > https://issues.apache.org/jira/browse/IMPALA-5903 was fixed every > query type has some kind of result set (even if it's just a status > message). impala-shell tries to be a bit more clever and report DML > results and similar in a special way, but other clients just fetch > the result set. > > If you look at ImpalaServer::FetchInternal() if called once the > query is in the EXCEPTION state it will end up raising the query > status as a BeeswaxException. > Yeah WARNING isn't a query state, it's just that the error log > mechanism can be used to return warnings that don't fail the query. > > It should be valid to call fetch on any query. Since > https://issues.apache.org/jira/browse/IMPALA-5903 was fixed every > query type has some kind of result set (even if it's just a status > message). impala-shell tries to be a bit more clever and report DML > results and similar in a special way, but other clients just fetch > the result set. > > If you look at ImpalaServer::FetchInternal() if called once the > query is in the EXCEPTION state it will end up raising the query > status as a BeeswaxException. HI Tim, Thanks a lot for the review and comments. I agree with you that calling fetch() would make the two queries end that the same state. However, adding the logic I mentioned earlier would complicate the wait_to_finish() method quite a bit. My thinking is that, we call get_warning_log() all over the places, even though we know some are ERROR and some are warning. So I propose adding a new method get_error_log() so we have two methods to choose from. To fix the jira here, we simply need to replace the one called when EXCEPTION state is detected with get_error_log. This way, the code looks clear, and we don't need to touch any other testcases I touched with previous rev. Would you please take a look at rev3 I just uploaded? Thanks a lot. -- To view, visit http://gerrit.cloudera.org:8080/12022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e Gerrit-Change-Number: 12022 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Wed, 12 Dec 2018 00:29:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12073 ) Change subject: IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3553/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12073 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Gerrit-Change-Number: 12073 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Wed, 12 Dec 2018 00:31:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5474: Adding a trivial subquery turns error into warning
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12022 ) Change subject: IMPALA-5474: Adding a trivial subquery turns error into warning .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3552/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e Gerrit-Change-Number: 12022 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Wed, 12 Dec 2018 00:31:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/12073 ) Change subject: IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12073 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Gerrit-Change-Number: 12073 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Wed, 12 Dec 2018 00:25:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12073 Change subject: IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" .. IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string" The fix for IMPALA-5929 introduced a bug that produced wrong results. This bug is detailed in IMPALA-7960. Reverting for now. This reverts commit 545163bb0a5c86aa02652d0557871f5b694a6c82. Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M testdata/workloads/functional-query/queries/QueryTest/exprs.test 3 files changed, 0 insertions(+), 100 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/12073/1 -- To view, visit http://gerrit.cloudera.org:8080/12073 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c Gerrit-Change-Number: 12073 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-5474: Adding a trivial subquery turns error into warning
Yongjun Zhang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12022 ) Change subject: IMPALA-5474: Adding a trivial subquery turns error into warning .. IMPALA-5474: Adding a trivial subquery turns error into warning After adding a subquery to a query that fails with ERROR, it fails with WARNING. The fix here makes it return ERROR. Testing: Added unit tests; Done real cluster testing with reported cases. Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 27 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/12022/3 -- To view, visit http://gerrit.cloudera.org:8080/12022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibedb11dd3d50bcdb21d508f7d21691925491946e Gerrit-Change-Number: 12022 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongjun Zhang
[Impala-ASF-CR] IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr.
Andrew Sherman has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12068 ) Change subject: IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. .. IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. These two classes evaluate scalar expressions. Previously codegen was done by calling ScalarExpr::GetCodegendComputeFnWrapper which generates a static method that calls the scalar expression evaluation methods. Make this more efficient by using cross-compiled code which is customized at codegen time. Add cross-compiled files is-not-empty-predicate-ir.cc and valid-tuple-id-ir.cc These files contain the methods that will be specialized during code generation These methods have to be static to match the expected method signatures Refactor the existing scalar methods so that they call a new internal static method which contains the implementation of the scalar expression This static method is called from both the old member function and the new cross-compiled method. IMPALA-7657 also requests replacing GetCodegendComputeFnWrapper() in TupleIsNullPredicate. In the current Impala code this method is never called. This is because TupleIsNullPredicate is always wrapped in an IfExpr. This is always codegen'd by IfExpr's GetCodegendComputeFnWrapper() method. There is a separate Jira IMPALA-7655 to improve codegen of IfExpr. Minor corrections: Correct the link to llvm tutorial in LlvmCodegen. Make a method private in TupleIsNullPredicate.java. TESTING: The changed scalar expressions are well exercised by current tests. Ran end-to-end tests. Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.h M be/src/exprs/CMakeLists.txt A be/src/exprs/is-not-empty-predicate-ir.cc M be/src/exprs/is-not-empty-predicate.cc M be/src/exprs/is-not-empty-predicate.h M be/src/exprs/slot-ref.cc A be/src/exprs/valid-tuple-id-ir.cc M be/src/exprs/valid-tuple-id.cc M be/src/exprs/valid-tuple-id.h M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java 12 files changed, 205 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/12068/2 -- To view, visit http://gerrit.cloudera.org:8080/12068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 Gerrit-Change-Number: 12068 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1579/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Dec 2018 00:14:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5200: Count child time for parent's total time
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11791 ) Change subject: IMPALA-5200: Count child time for parent's total time .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3551/ -- To view, visit http://gerrit.cloudera.org:8080/11791 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e Gerrit-Change-Number: 11791 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 12 Dec 2018 00:10:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 5: (11 comments) http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@25 PS5, Line 25: d flake8: E501 line too long (162 > 90 characters) http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@29 PS5, Line 29: from thrift.protocol import TCompactProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@30 PS5, Line 30: from thrift.TSerialization import deserialize flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@31 PS5, Line 31: from RuntimeProfile.ttypes import TRuntimeProfileTree flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@33 PS5, Line 33: import argparse flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@34 PS5, Line 34: import base64 flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@35 PS5, Line 35: import datetime flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@36 PS5, Line 36: import sys flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@37 PS5, Line 37: import zlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@39 PS5, Line 39: import matplotlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/5/bin/plot_profile_resource_usage.py@41 PS5, Line 41: import matplotlib.pyplot as plt flake8: E402 module level import not at top of file -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 23:39:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Hello Michael Ho, Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12069 to look at the new patch set (#5). Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. Example: https://user-images.githubusercontent.com/151514/49830685-bb7efd80-fd46-11e8-8e23-9f5bc47635c1.png This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 874 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/5 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1578/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 23:26:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 4: (11 comments) http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@25 PS4, Line 25: d flake8: E501 line too long (162 > 90 characters) http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@29 PS4, Line 29: from thrift.protocol import TCompactProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@30 PS4, Line 30: from thrift.TSerialization import deserialize flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@31 PS4, Line 31: from RuntimeProfile.ttypes import TRuntimeProfileTree flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@33 PS4, Line 33: import argparse flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@34 PS4, Line 34: import base64 flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@35 PS4, Line 35: import datetime flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@36 PS4, Line 36: import sys flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@37 PS4, Line 37: import zlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@39 PS4, Line 39: import matplotlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/4/bin/plot_profile_resource_usage.py@41 PS4, Line 41: import matplotlib.pyplot as plt flake8: E402 module level import not at top of file -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 22:55:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1577/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 23:12:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1576/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 23:10:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12069 to look at the new patch set (#4). Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. Example: https://user-images.githubusercontent.com/151514/49830685-bb7efd80-fd46-11e8-8e23-9f5bc47635c1.png This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 874 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/4 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Lars Volker has removed Ambreen Kazi from this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Removed reviewer Ambreen Kazi. -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@43 PS3, Line 43: class ResourceUsageNotFound(Exception): > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 22:55:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7212: Remove dead code data-stream-mgr.cc
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12064 ) Change subject: IMPALA-7212: Remove dead code data-stream-mgr.cc .. IMPALA-7212: Remove dead code data-stream-mgr.cc Dead code which was accidentally left out in the last patch of IMPALA-7212. Testing done: Built Impala debug and release builds Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542 Reviewed-on: http://gerrit.cloudera.org:8080/12064 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- D be/src/runtime/data-stream-mgr.cc 1 file changed, 0 insertions(+), 298 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542 Gerrit-Change-Number: 12064 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7212: Remove dead code data-stream-mgr.cc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12064 ) Change subject: IMPALA-7212: Remove dead code data-stream-mgr.cc .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542 Gerrit-Change-Number: 12064 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 22:44:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12069 )
Change subject: IMPALA-7694: Add host resource usage metrics to profile
..
Patch Set 3:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py
File bin/plot_profile_resource_usage.py:
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@25
PS1, Line 25: d
> flake8: E501 line too long (162 > 90 characters)
Ignoring this for readability
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@28
PS1, Line 28:
> flake8: E402 module level import not at top of file
Ignoring this and the others.
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@42
PS1, Line 42:
> flake8: F401 'collections.namedtuple' imported but unused
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@44
PS1, Line 44: pass
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@47
PS1, Line 47: class CountersNotFound(Exception):
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@50
PS1, Line 50:
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@53
PS1, Line 53:
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@64
PS1, Line 64: return parser
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@80
PS1, Line 80: raise ResourceUsageNotFound("Profile does not contain
resource resource usage.")
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@97
PS1, Line 97: if not per_node_counters:
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@101
PS1, Line 101:
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@129
PS1, Line 129:
> flake8: E226 missing whitespace around arithmetic operator
Done
http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@148
PS1, Line 148: tree = TRuntimeProfileTree()
> flake8: W391 blank line at end of file
Done
http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@370
PS2, Line 370:
> flake8: E124 closing bracket does not match visual indentation
Done
http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@385
PS2, Line 385: l
> flake8: E124 closing bracket does not match visual indentation
Done
http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@414
PS2, Line 414:
> flake8: F821 undefined name 'result'
Done
http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@432
PS2, Line 432: h
> flake8: E713 test for membership should be 'not in'
Done
--
To view, visit http://gerrit.cloudera.org:8080/12069
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9
Gerrit-Change-Number: 12069
Gerrit-PatchSet: 3
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Comment-Date: Tue, 11 Dec 2018 22:37:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 3: (12 comments) http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@25 PS3, Line 25: d flake8: E501 line too long (162 > 90 characters) http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@29 PS3, Line 29: from thrift.protocol import TCompactProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@30 PS3, Line 30: from thrift.TSerialization import deserialize flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@31 PS3, Line 31: from RuntimeProfile.ttypes import TRuntimeProfileTree flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@33 PS3, Line 33: import argparse flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@34 PS3, Line 34: import base64 flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@35 PS3, Line 35: import datetime flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@36 PS3, Line 36: import sys flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@37 PS3, Line 37: import zlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@39 PS3, Line 39: import matplotlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@41 PS3, Line 41: import matplotlib.pyplot as plt flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/3/bin/plot_profile_resource_usage.py@43 PS3, Line 43: class ResourceUsageNotFound(Exception): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 22:38:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12069 to look at the new patch set (#3). Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. Example: https://user-images.githubusercontent.com/151514/49830685-bb7efd80-fd46-11e8-8e23-9f5bc47635c1.png This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 873 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/3 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 2: (16 comments) http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@25 PS2, Line 25: d flake8: E501 line too long (162 > 90 characters) http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@29 PS2, Line 29: from thrift.protocol import TCompactProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@30 PS2, Line 30: from thrift.TSerialization import deserialize flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@31 PS2, Line 31: from RuntimeProfile.ttypes import TRuntimeProfileTree flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@33 PS2, Line 33: import argparse flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@34 PS2, Line 34: import base64 flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@35 PS2, Line 35: import datetime flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@36 PS2, Line 36: import sys flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@37 PS2, Line 37: import zlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@39 PS2, Line 39: import matplotlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@41 PS2, Line 41: import matplotlib.pyplot as plt flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/2/bin/plot_profile_resource_usage.py@43 PS2, Line 43: class ResourceUsageNotFound(Exception): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@370 PS2, Line 370: ] flake8: E124 closing bracket does not match visual indentation http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@385 PS2, Line 385: ] flake8: E124 closing bracket does not match visual indentation http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@414 PS2, Line 414: r flake8: F821 undefined name 'result' http://gerrit.cloudera.org:8080/#/c/12069/2/tests/query_test/test_observability.py@432 PS2, Line 432: n flake8: E713 test for membership should be 'not in' -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 22:35:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12069 to look at the new patch set (#2). Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. Example: https://user-images.githubusercontent.com/151514/49830685-bb7efd80-fd46-11e8-8e23-9f5bc47635c1.png This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 875 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/2 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-4555: Make QueryState's status reporting more robust
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12049 ) Change subject: IMPALA-4555: Make QueryState's status reporting more robust .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/12049/1/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/12049/1/be/src/runtime/query-state.cc@58 PS1, Line 58: DEFINE_int32(status_report_max_failures, 3, : "Max number of consecutive failed status reports to allow before cancelling"); > I thought we want to use a fixed timeout approach for the maximum retries ? Actually, max_retries seems to be safer in the sense that it guarantees a minimum amount of time the thread will sleep before giving up as we know the sleep time between each retry. With an absolute timeout, there is no guarantee on the number of retries we will do. If the system is overloaded, the query state thread may not get to run very often before expiration so the number of retries is non-deterministic. http://gerrit.cloudera.org:8080/#/c/12049/1/be/src/runtime/query-state.cc@368 PS1, Line 368: fis_map_[id]->runtime_state()->ClearUnreportedErrors(); There is a race here: we may be clearing newly added errors we added after the profile was computed. -- To view, visit http://gerrit.cloudera.org:8080/12049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib6007013fc2c9e8eeba11b752ee58fb3038da971 Gerrit-Change-Number: 12049 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 11 Dec 2018 22:25:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7728: [DOCS] Added a section on Changing Privileges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12071 ) Change subject: IMPALA-7728: [DOCS] Added a section on Changing Privileges .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/175/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I955cb49cae24be6a93a90ccb5f2aa6ceb29cee8b Gerrit-Change-Number: 12071 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 22:19:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7728: [DOCS] Added a section on Changing Privileges
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12071 Change subject: IMPALA-7728: [DOCS] Added a section on Changing Privileges .. IMPALA-7728: [DOCS] Added a section on Changing Privileges Change-Id: I955cb49cae24be6a93a90ccb5f2aa6ceb29cee8b --- M docs/topics/impala_authorization.xml 1 file changed, 179 insertions(+), 128 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/12071/1 -- To view, visit http://gerrit.cloudera.org:8080/12071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I955cb49cae24be6a93a90ccb5f2aa6ceb29cee8b Gerrit-Change-Number: 12071 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] Fix catalog port description in docs
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12070 ) Change subject: Fix catalog port description in docs .. Fix catalog port description in docs Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 Reviewed-on: http://gerrit.cloudera.org:8080/12070 Tested-by: Impala Public Jenkins Reviewed-by: Alex Rodoni --- M docs/topics/impala_ports.xml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Verified Alex Rodoni: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 Gerrit-Change-Number: 12070 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Fix catalog port description in docs
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12070 ) Change subject: Fix catalog port description in docs .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 Gerrit-Change-Number: 12070 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 22:12:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7728: [DOCS] Added a section on Changing Privileges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12071 ) Change subject: IMPALA-7728: [DOCS] Added a section on Changing Privileges .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/175/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I955cb49cae24be6a93a90ccb5f2aa6ceb29cee8b Gerrit-Change-Number: 12071 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 22:10:11 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix catalog port description in docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12070 ) Change subject: Fix catalog port description in docs .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/174/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 Gerrit-Change-Number: 12070 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 21:49:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1575/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 21:46:52 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix catalog port description in docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12070 ) Change subject: Fix catalog port description in docs .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/174/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 Gerrit-Change-Number: 12070 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 21:45:35 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix catalog port description in docs
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12070 Change subject: Fix catalog port description in docs .. Fix catalog port description in docs Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 --- M docs/topics/impala_ports.xml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/12070/1 -- To view, visit http://gerrit.cloudera.org:8080/12070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I453e3a4d930e4f948bac81ddba9c8e096d67a326 Gerrit-Change-Number: 12070 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12068 ) Change subject: IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1574/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 Gerrit-Change-Number: 12068 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 11 Dec 2018 21:37:28 + Gerrit-HasComments: No
[Impala-ASF-CR] Update version to 3.2.0-SNAPSHOT
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/12053 ) Change subject: Update version to 3.2.0-SNAPSHOT .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69547de6e768470820930fe05f444df416c5f1de Gerrit-Change-Number: 12053 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Tue, 11 Dec 2018 21:32:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12069 Change subject: IMPALA-7694: Add host resource usage metrics to profile .. IMPALA-7694: Add host resource usage metrics to profile This change adds a mechanism to collect host resource usage metrics to profiles. Metric collection can be controlled through a new query option 'RESOURCE_TRACE_RATIO'. It specifies the probability with which metrics collection will be enabled. Collection always happens per query for all executors that run one or more fragment instances of the query. This mechanism adds a new time series counter class that collects all measured values and does not re-sample them. It will re-sample values when printing them into a string profile to a max of 64 values, but Thrift profiles will contain the full list of values. We add a new section "Per Node Resource Usage" to the profile to store and show these values: Per Node Resource Usage: lv-desktop:22000: CpuIoWaitPercentage (500.000ms): 0, 0 CpuSysPercentage (500.000ms): 1, 1 CpuUserPercentage (500.000ms): 4, 0 - ScratchBytesRead: 0 - ScratchBytesWritten: 0 - ScratchFileUsedBytes: 0 - ScratchReads: 0 (0) - ScratchWrites: 0 (0) - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns This change also uses the aforementioned mechanism to collect CPU usage metrics (user, system, and IO wait time). This change also adds a tool to decode a Thrift profile and plot the contained usage metrics using matplotlib. This change also exposes the scratch profile within the per node resource usage section. Testing: This change contains a unit test for the system level metrics collection and e2e tests for the profile changes. Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/CMakeLists.txt M be/src/util/periodic-counter-updater.cc M be/src/util/periodic-counter-updater.h M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h A be/src/util/system-state-info-test.cc A be/src/util/system-state-info.cc A be/src/util/system-state-info.h A bin/plot_profile_resource_usage.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Metrics.thrift M common/thrift/RuntimeProfile.thrift M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 29 files changed, 873 insertions(+), 94 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12069/1 -- To view, visit http://gerrit.cloudera.org:8080/12069 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3aedc20c553ab8d7ed50f72a1a936eba151487d9 Gerrit-Change-Number: 12069 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-7694: Add host resource usage metrics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12069 ) Change subject: IMPALA-7694: Add host resource usage metrics to profile .. Patch Set 1: (27 comments) http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py File bin/plot_profile_resource_usage.py: http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@25 PS1, Line 25: d flake8: E501 line too long (162 > 90 characters) http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@28 PS1, Line 28: from thrift.protocol import TCompactProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@29 PS1, Line 29: from thrift.TSerialization import deserialize flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@30 PS1, Line 30: from RuntimeProfile.ttypes import TRuntimeProfileTree flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@32 PS1, Line 32: import argparse flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@33 PS1, Line 33: import base64 flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@34 PS1, Line 34: import datetime flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@35 PS1, Line 35: import sys flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@36 PS1, Line 36: import zlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@38 PS1, Line 38: import matplotlib flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@40 PS1, Line 40: import matplotlib.pyplot as plt flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@42 PS1, Line 42: from collections import namedtuple flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@42 PS1, Line 42: from collections import namedtuple flake8: F401 'collections.namedtuple' imported but unused http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@44 PS1, Line 44: class ResourceUsageNotFound(Exception): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@47 PS1, Line 47: class CountersNotFound(Exception): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@50 PS1, Line 50: class CounterNotFound(Exception): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@53 PS1, Line 53: def create_parser(): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@64 PS1, Line 64: def find_resource_usage_nodes(thrift_profile): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@80 PS1, Line 80: def extract_ts_counters(thrift_profile): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@97 PS1, Line 97: def byte_to_percent(values): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@101 PS1, Line 101: def plot_counter_for_all_hosts(per_node_counters, counter_name, output_file): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@129 PS1, Line 129: / flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/12069/1/bin/plot_profile_resource_usage.py@148 PS1, Line 148: flake8: W391 blank line at end of file http://gerrit.cloudera.org:8080/#/c/12069/1/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/12069/1/tests/query_test/test_observability.py@370 PS1, Line 370: ] flake8: E124 closing bracket does not match visual indentation http://gerrit.cloudera.org:8080/#/c/12069/1/tests/query_test/test_observability.py@385 PS1, Line 385: ] flake8: E124 closing bracket does not match visual indentation
[Impala-ASF-CR] IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12068 Change subject: IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. .. IMPALA-7657: Codegen IsNotEmptyPredicate and ValidTupleIdExpr. These two classes evaluate scalar expressions. Previously codegen was done by calling ScalarExpr::GetCodegendComputeFnWrapper which generates a static method that calls the scalar expression evaluation methods. Make this more efficient by using cross-compiled code which is customized at codegen time. Add cross-compiled files is-not-empty-predicate-ir.cc and valid-tuple-id-ir.cc These files contain the methods that will be specialized during code generation These methods have to be static to match the expected method signatures Refactor the existing scalar methods so that they call a new internal static method which contains the implementation of the scalar expression This static method is called from both the old member function and the new cross-compiled method. IMPALA-7657 also requests replacing GetCodegendComputeFnWrapper() in TupleIsNullPredicate. In the current Impala code this method is never called. This is because TupleIsNullPredicate is always wrapped in an IfExpr. This is always codegen'd by IfExpr's GetCodegendComputeFnWrapper() method. There is a separate Jira IMPALA-7655 to improve codegen of IfExpr. Minor corrections: Correct the link to llvm tutorial in LlvmCodegen. Make a method private in TupleIsNullPredicate.java. TESTING: The changed scalar expressions are well exercised by current tests. Ran end-to-end tests. Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.h M be/src/exprs/CMakeLists.txt A be/src/exprs/is-not-empty-predicate-ir.cc M be/src/exprs/is-not-empty-predicate.cc M be/src/exprs/is-not-empty-predicate.h M be/src/exprs/slot-ref.cc A be/src/exprs/valid-tuple-id-ir.cc M be/src/exprs/valid-tuple-id.cc M be/src/exprs/valid-tuple-id.h M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java 12 files changed, 205 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/12068/1 -- To view, visit http://gerrit.cloudera.org:8080/12068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifb87b9e3b879c278ce8638d97bcb320a7555a6b3 Gerrit-Change-Number: 12068 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7939: Fix issue where CTE is categorized as DML statement
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12052 ) Change subject: IMPALA-7939: Fix issue where CTE is categorized as DML statement .. IMPALA-7939: Fix issue where CTE is categorized as DML statement The logic that checks whether a CTE is DML or SELECT uses shlex that splits the statement into tokens and check if any of the tokens matches the DML regular expression. Before this patch, the shlex was set to posix=True, which means the quotes are stripped from the token, e.g. select a from foo where a = 'update' becomes ['select', 'a', 'from', 'foo', 'where', 'a', '=', 'update']. As a result, any token that contains "insert", "delete", "upsert", and "update" in it will be categorized as DML even though the token is part of string literal value. This patch fixes the issue by setting posix=False in shlex that preserves the quotes. For example: ['select', 'a', 'from', 'foo', 'where', 'a', '=', '"update"'] Testing: - Added a new shell test - Ran all shell tests Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Reviewed-on: http://gerrit.cloudera.org:8080/12052 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 16 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Gerrit-Change-Number: 12052 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7939: Fix issue where CTE is categorized as DML statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12052 ) Change subject: IMPALA-7939: Fix issue where CTE is categorized as DML statement .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Gerrit-Change-Number: 12052 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 20:57:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5200: Count child time for parent's total time
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11791 ) Change subject: IMPALA-5200: Count child time for parent's total time .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1573/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11791 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e Gerrit-Change-Number: 11791 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 20:50:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5200: Count child time for parent's total time
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11791 ) Change subject: IMPALA-5200: Count child time for parent's total time .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3551/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11791 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e Gerrit-Change-Number: 11791 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 20:15:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5200: Count child time for parent's total time
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11791 ) Change subject: IMPALA-5200: Count child time for parent's total time .. Patch Set 3: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/11791 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e Gerrit-Change-Number: 11791 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 20:15:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5200: Count child time for parent's total time
Hello Philip Zeyliger, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11791 to look at the new patch set (#3). Change subject: IMPALA-5200: Count child time for parent's total time .. IMPALA-5200: Count child time for parent's total time One problem with the total time counter on runtime profiles is that a parent's time may not be updated if execution is stuck in a child node. The child can accumulate time while the parent is stuck at zero. This leads to incorrect or misleading calculations of total time or non-child time for the parent node during execution. This makes a modest change in calculation for total time for parent nodes. It takes advantage of the fact that the parent should count all of the time from all of its children as total time for itself. Specifically, if a parent has accumulated X in its total timer and its children have accumulated Y summed across all of their timers, then a parent's total time should be at least max(X, Y). There is no way to know the appropriate overlap between X and Y, so this uses a conservative calculation assuming complete overlap. This prevents a parent node from reporting itself as 100% non-child time when it is actually stuck executing child code. However, it does not help if a child node is stuck and is not reporting its own time. Testing: - Added test case to runtime-profile-test - Core tests pass Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e --- M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h 3 files changed, 125 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/11791/3 -- To view, visit http://gerrit.cloudera.org:8080/11791 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e Gerrit-Change-Number: 11791 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5200: Count child time for parent's total time
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11791 ) Change subject: IMPALA-5200: Count child time for parent's total time .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11791/2/be/src/util/runtime-profile.cc File be/src/util/runtime-profile.cc: http://gerrit.cloudera.org:8080/#/c/11791/2/be/src/util/runtime-profile.cc@427 PS2, Line 427: total_time_ns_ = max(children_total_time, total_time_counter()->value()); > Might be worth mentioning this JIRA in a comment here to justify taking the Added a comment with the JIRA and a brief explanation. -- To view, visit http://gerrit.cloudera.org:8080/11791 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c1191c39fd18b6be45325366a74cf54908c77e Gerrit-Change-Number: 11791 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 20:14:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7212: Remove dead code data-stream-mgr.cc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12064 ) Change subject: IMPALA-7212: Remove dead code data-stream-mgr.cc .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3550/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542 Gerrit-Change-Number: 12064 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 18:52:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7212: Remove dead code data-stream-mgr.cc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12064 ) Change subject: IMPALA-7212: Remove dead code data-stream-mgr.cc .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542 Gerrit-Change-Number: 12064 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 11 Dec 2018 18:52:57 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: WIP: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1572/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 18:20:56 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: WIP: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/12065/1/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/12065/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1556 PS1, Line 1556: scalar_reader_map_[node->col_idx] = static_cast(col_reader); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/12065/1/be/src/util/dict-encoding.h File be/src/util/dict-encoding.h: http://gerrit.cloudera.org:8080/#/c/12065/1/be/src/util/dict-encoding.h@549 PS1, Line 549: int64_t num_to_skip = std::min(num_literal_values_ - next_literal_idx_, num_remaining); line too long (101 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Dec 2018 17:46:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: IMPALA-5843: Use page index in Parquet files to skip pages
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12065 Change subject: WIP: IMPALA-5843: Use page index in Parquet files to skip pages .. WIP: IMPALA-5843: Use page index in Parquet files to skip pages Initial prototype of page filtering. Some detailes are smelly, but conceptually it is getting into shape. The read and evaluation of the page index is done by the HdfsParquetScanner. At first, we determine the row ranges we are interested in, and based on the row ranges we determine the filtered pages for each column that we are reading. We still issue one ScanRange per column chunk, but we specify sub-ranges that store the filtered pages, i.e. we don't read the whole column chunk, but only fractions of it. Pages are not aligned across column chunks, i.e. page #2 of column A might store completely different rows than page #2 of column B. It means we need to implement some kind of row-skipping logic when we read the data pages. This logic is implemented in BaseScalarColumnReader and ScalarColumnReader. Collection column readers know nothing about page filtering. I also extended the decoders with value-skipping functionalities. TODOs: * add unit tests (BE tests) wherever possible * fix the smelly code parts * implement row-skipping in MaterializeValueBatchRepeatedDefLevel() * add counters about filtered pages * generate files by Impala and Parquet-MR for EE tests * test with nested types * performance measurements Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-bool-decoder.cc M be/src/exec/parquet/parquet-bool-decoder.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-level-decoder.h M be/src/exprs/literal.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/bit-stream-utils.h M be/src/util/bit-stream-utils.inline.h M be/src/util/dict-encoding.h M be/src/util/rle-encoding.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift 22 files changed, 790 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12065/1 -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7939: Fix issue where CTE is categorized as DML statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12052 ) Change subject: IMPALA-7939: Fix issue where CTE is categorized as DML statement .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Gerrit-Change-Number: 12052 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 17:00:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7939: Fix issue where CTE is categorized as DML statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12052 ) Change subject: IMPALA-7939: Fix issue where CTE is categorized as DML statement .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3549/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Gerrit-Change-Number: 12052 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 17:00:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7939: Fix issue where CTE is categorized as DML statement
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12052 ) Change subject: IMPALA-7939: Fix issue where CTE is categorized as DML statement .. Patch Set 7: > Patch Set 7: Code-Review+2 > > Can you file a JIRA to figure out how to remove this client-side code? https://issues.apache.org/jira/browse/IMPALA-7956 -- To view, visit http://gerrit.cloudera.org:8080/12052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Gerrit-Change-Number: 12052 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 16:59:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7939: Fix issue where CTE is categorized as DML statement
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12052 ) Change subject: IMPALA-7939: Fix issue where CTE is categorized as DML statement .. Patch Set 7: Code-Review+2 Can you file a JIRA to figure out how to remove this client-side code? -- To view, visit http://gerrit.cloudera.org:8080/12052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7 Gerrit-Change-Number: 12052 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 11 Dec 2018 16:50:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1570/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 15:23:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1571/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 15:33:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. IMPALA-6591: Fix test_ssl flaky test test_ssl has a logic that waits for the number of in-flight queries to be 1. However, the logic for wait_for_num_in_flight_queries(1) only waits for the condition to be true for a period of time and does not throw an exception when the time has elapsed and the condition is not met. In other words, the logic in test_ssl that loops while the number of in-flight queries is 1 never gets executed. I was able to simulate this issue by making Impala shell start much longer. Prior to this patch, in the event that Impala shell took much longer to start, the test started sending the commands to Impala shell even when Impala shell was not ready to receive commands. The patch fixes the issue by waiting until Impala shell is connected. The patch also adds assert in other places that calls wait_for_num_in_flight_queries and updates the default behavior for Impala shell to wait until it is connected. Testing: - Ran core and exhaustive tests several times on CentOS 6 without any issue Change-Id: I9805269d8b806aecf5d744c219967649a041d49f --- M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 6 files changed, 41 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/12047/6 -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/12047/5/tests/shell/util.py File tests/shell/util.py: http://gerrit.cloudera.org:8080/#/c/12047/5/tests/shell/util.py@88 PS5, Line 88: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/12047/5/tests/shell/util.py@105 PS5, Line 105: > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 14:59:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/12047/5/tests/shell/util.py File tests/shell/util.py: http://gerrit.cloudera.org:8080/#/c/12047/5/tests/shell/util.py@88 PS5, Line 88: def run_impala_shell_cmd(shell_args, expect_success=True, stdin_input=None, flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/12047/5/tests/shell/util.py@105 PS5, Line 105: def run_impala_shell_cmd_no_expect(shell_args, stdin_input=None, flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 14:50:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. IMPALA-6591: Fix test_ssl flaky test test_ssl has a logic that waits for the number of in-flight queries to be 1. However, the logic for wait_for_num_in_flight_queries(1) only waits for the condition to be true for a period of time and does not throw an exception when the time has elapsed and the condition is not met. In other words, the logic in test_ssl that loops while the number of in-flight queries is 1 never gets executed. I was able to simulate this issue by making Impala shell start much longer. Prior to this patch, in the event that Impala shell took much longer to start, the test started sending the commands to Impala shell even when Impala shell was not ready to receive commands. The patch fixes the issue by waiting until Impala shell is connected. The patch also adds assert in other places that calls wait_for_num_in_flight_queries and updates the default behavior for Impala shell to wait until it is connected. Testing: - Ran core and exhaustive tests several times on CentOS 6 without any issue Change-Id: I9805269d8b806aecf5d744c219967649a041d49f --- M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 6 files changed, 37 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/12047/5 -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-6591: Fix test ssl flaky test
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12047 ) Change subject: IMPALA-6591: Fix test_ssl flaky test .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/12047/2/tests/custom_cluster/test_client_ssl.py File tests/custom_cluster/test_client_ssl.py: http://gerrit.cloudera.org:8080/#/c/12047/2/tests/custom_cluster/test_client_ssl.py@88 PS2, Line 88: p = ImpalaShell(args="--ssl") > How many places would these be? I think the behavior should really be wait Done http://gerrit.cloudera.org:8080/#/c/12047/4/tests/shell/util.py File tests/shell/util.py: http://gerrit.cloudera.org:8080/#/c/12047/4/tests/shell/util.py@128 PS4, Line 128: or will wait until : Impala shell is connected for the specified timeout unless wait_util_connected is : set to False o > I don't think this second sentence is necessary. Done http://gerrit.cloudera.org:8080/#/c/12047/4/tests/shell/util.py@132 PS4, Line 132: self.shell_process = self._start_new_shell_process(args, env=env) > This should have a timeout. It can be high (5min) but that will make a hang Updated the CR but using 1 minute as the default. Done. -- To view, visit http://gerrit.cloudera.org:8080/12047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9805269d8b806aecf5d744c219967649a041d49f Gerrit-Change-Number: 12047 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Dec 2018 14:49:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7914: Base class for statement-like AST nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12018 ) Change subject: IMPALA-7914: Base class for statement-like AST nodes .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12018 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie565ff02ad74f805a667017ba9bc8c0a2697a97b Gerrit-Change-Number: 12018 Gerrit-PatchSet: 7 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Tue, 11 Dec 2018 09:53:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7914: Base class for statement-like AST nodes
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12018 ) Change subject: IMPALA-7914: Base class for statement-like AST nodes .. IMPALA-7914: Base class for statement-like AST nodes In order to integrate expression rewrites into the analysis phase, the expression analyze() operation must be able to replace one expression node with another. Statements, however, are analyzed in place. The two types of parse nodes thus need different analyze() semantics. To prepare for that goal, this patch introduces a new StmtNode class as the base for all statement-like AST nodes. The existing analyze() method moves to StmtNode. While Expr still defines this method for now, the future goal is to change the Expr analyze() semantics. Tests: This is purely a code restructuring, no functional changes. Reran all FE tests. Change-Id: Ie565ff02ad74f805a667017ba9bc8c0a2697a97b Reviewed-on: http://gerrit.cloudera.org:8080/12018 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/FunctionArgs.java M fe/src/main/java/org/apache/impala/analysis/HdfsCachingOp.java M fe/src/main/java/org/apache/impala/analysis/KuduPartitionParam.java M fe/src/main/java/org/apache/impala/analysis/ParseNode.java M fe/src/main/java/org/apache/impala/analysis/PartitionDef.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpecBase.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/RangePartition.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java A fe/src/main/java/org/apache/impala/analysis/StmtNode.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TableSampleClause.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java 18 files changed, 72 insertions(+), 21 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12018 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie565ff02ad74f805a667017ba9bc8c0a2697a97b Gerrit-Change-Number: 12018 Gerrit-PatchSet: 8 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
