[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 8: The reason the build failed is because of "test_user". This user is not considered valid in the cluster tests because it does not belong to a group. I updated the CustomClusterGroupMapping to add new users and groups for the owner_privilege tests. This removes previous concerns about running with user "root". No core code was changed for patch set 8. -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 8 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 07:00:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Adam Holley has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. IMPALA-7503: SHOW GRANT USER not showing all privileges. This patch fixes the SHOW GRANT USER statement to show all privileges granted to a user, either directly via object ownership, or granted through a role via a group the user belongs to. The output for SHOW GRANT USER will have two additional columns for privilege name and privilege type so the user can know where the privilege comes from. Truncated sample showing two columns that are different from role: ++++--+-... | principal_type | principal_name | scope | database | ... ++++--+-... | USER | foo| table | foo_db | ... | ROLE | foo_role | server | | ... ++++--+-... Testing: - Create new custom cluster test with custom group mapping. - Ran FE and custom cluster tests. Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 --- M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/catalog/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java A fe/src/test/java/org/apache/impala/service/CustomClusterGroupMapper.java A fe/src/test/java/org/apache/impala/service/CustomClusterResourceAuthorizationProvider.java M fe/src/test/resources/sentry-site_no_oo.xml.template M fe/src/test/resources/sentry-site_oo.xml.template M fe/src/test/resources/sentry-site_oo_nogrant.xml.template M testdata/bin/run-sentry-service.sh A testdata/workloads/functional-query/queries/QueryTest/show_grant_user.test M tests/authorization/test_grant_revoke.py M tests/authorization/test_owner_privileges.py A tests/authorization/test_show_grant_user.py M tests/common/sentry_cache_test_suite.py 15 files changed, 1,400 insertions(+), 177 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11531/9 -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 9 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3241/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 9 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 07:02:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11539 Change subject: IMPALA-7599: make the number of local cache retries configurable .. IMPALA-7599: make the number of local cache retries configurable Under heavy read/write load, the number of retries needed for queries in order to skip over inconsistent metadata exceptions needs to be set higher. This change makes the number of retries configurable. It can be set with the newly added flag --local_catalog_max_retries. In addition, this change increases the default from 10 to 40, which was sufficient when handling several workloads with high read/write load. Made several fixes to exception messages. Testing: - manual tests - added an e2e test that sets the flag and checks for inconsistent metadata Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_local_catalog.py 7 files changed, 66 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/1 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Hello Bharath Vissapragada, Tianyi Wang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11534 to look at the new patch set (#3). Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. IMPALA-7622: adds profile metrics when fetching incremental stats When computing incremental statistics by fetching the stats directly from catalogd, a potentially expensive RPC is made from the impalad coordinator to catalogd. This change adds metrics to the frontend section of the profile to track how long the request takes, the size of the compressed bytes received, and the number of partitions received. The profile for a 'compute incremental ...' command on a table with no statistics looks like this: Frontend: - StatsFetch.CompressedBytes: 0 - StatsFetch.TotalPartitions: 24 - StatsFetch.NumPartitionsWithStats: 0 - StatsFetch.Time: 26ms And the profile looks as follows when the table has stats, so the stats are fetched: Frontend: - StatsFetch.CompressedBytes: 24622 - StatsFetch.TotalPartitions: 23 - StatsFetch.NumPartitionsWithStats: 23 - StatsFetch.Time: 14ms Testing: - manual inspection - e2e test to check the profile Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_pull_stats.py 3 files changed, 93 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/11534/3 -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11534 ) Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11534/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java: http://gerrit.cloudera.org:8080/#/c/11534/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@122 PS2, Line 122: private static final String STATS_FETCH_TIME = STATS_FETCH_PREFIX + ".Time"; : private static final String STATS_FETCH_COMPRESSED_BYTES = : STATS_FETCH_PREFIX + ".CompressedBytes"; : private static final String STATS_FETCH_TOTAL_PARTITIONS = : STATS_FETCH_PREFIX + ".TotalPartitions"; : private static final String STATS_FETCH_NUM_PARTITIONS_WITH_STATS = : STATS_FETCH_PREFIX + ".NumPartitionsWithStats"; > Sorry missed this in the first round. Would be nice to add one-liners that Done -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 07:25:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/859/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 8 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 07:31:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11534 ) Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. Patch Set 3: Code-Review+2 carry +2 -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 07:38:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 9 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 07:46:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11534 ) Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/862/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 07:46:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11534 ) Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3242/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 07:47:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 07:55:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7595: Check the validity of the time part of Parquet timestamps
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11521 ) Change subject: IMPALA-7595: Check the validity of the time part of Parquet timestamps .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/11521/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11521/3//COMMIT_MSG@30 PS3, Line 30: that separate error message is not necessary. Yeah, but the current error message is a bit misleading IMO, since currently it says something like "dates should be between 1400-01-01 and -12-31". Or, maybe we can just change the error message to make it more general. http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/exec/parquet-column-readers.cc File be/src/exec/parquet-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/exec/parquet-column-readers.cc@697 PS3, Line 697: !TimestampValue::IsValidDate(val->date()) : || !TimestampValue::IsValidTime(val->time()) Would it make sense to merge these two functions and have a single TimestampValue::IsValid() member function instead? http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/runtime/timestamp-value.h File be/src/runtime/timestamp-value.h: http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/runtime/timestamp-value.h@195 PS3, Line 195: then nit: than -- To view, visit http://gerrit.cloudera.org:8080/11521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 Gerrit-Change-Number: 11521 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 09:52:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Hello Greg Rahn, Zoltan Borok-Nagy, Attila Jeges, Todd Lipcon, Tim Armstrong,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10950
to look at the new patch set (#16).
Change subject: IMPALA-376: add built-in functions for parsing JSON
..
IMPALA-376: add built-in functions for parsing JSON
This patch implements the same function as Hive UDF get_json_object.
We reuse RapidJson to parse the json string. In order to track the
memory used in RapidJson, we wrap FunctionContext into an allocator.
get_json_object accepts two parameters: a json string and a selector
(json path). We parse the json string into a Document tree and then
perform BFS according to the selector. For example, to process
get_json_object('[{\"a\":1}, {\"a\":2}, {\"a\":3}]', '$[*].a'),
we first perform '$[*]' to extract all the items in the root array.
Then we get a queue consists of {a:1},{a:2},{a:3} and perform '.a'
selector on all values in the queue. The final results is 1,2,3 in the
queue. As there're multiple results, they should be encapsulated into
an array. The output results is a string of '[1,2,3]'.
More examples can be found in expr-test.cc.
Test:
* Add unit tests in expr-test
* Add e2e tests in exprs.test
* Add tests in test_alloc_fail.py to check handling of out of memory
Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f
---
M be/src/exprs/CMakeLists.txt
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
A be/src/exprs/string-functions.cc
M be/src/exprs/string-functions.h
M be/src/util/string-util.cc
M be/src/util/string-util.h
M common/function-registry/impala_functions.py
M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-init.test
M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
11 files changed, 657 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/10950/16
--
To view, visit http://gerrit.cloudera.org:8080/10950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f
Gerrit-Change-Number: 10950
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/10950 )
Change subject: IMPALA-376: add built-in functions for parsing JSON
..
Patch Set 16:
(6 comments)
Thanks for your patience too. Adjusted the patch based on your comments.
http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/exprs/string-functions.cc
File be/src/exprs/string-functions.cc:
http://gerrit.cloudera.org:8080/#/c/10950/9/be/src/exprs/string-functions.cc@51
PS9, Line 51: static const bool kNeedFree = false;
> That memory should be tracked indirectly - results_pool_ is a FreePool, whi
Oh, yes. I thought it's the MemPool in udf.cc. Then I have no concerns :)
http://gerrit.cloudera.org:8080/#/c/10950/15/be/src/exprs/string-functions.cc
File be/src/exprs/string-functions.cc:
http://gerrit.cloudera.org:8080/#/c/10950/15/be/src/exprs/string-functions.cc@72
PS15, Line 72: if (originalSize >= newSize) return originalPtr;
> Put conditional on a single line.
Done
http://gerrit.cloudera.org:8080/#/c/10950/15/be/src/exprs/string-functions.cc@143
PS15, Line 143: } else { // multiple selected items, return an array string
> Can't we use RETURN_NULL_IF_OOM in some of these places?
Yes, missing these. Done
http://gerrit.cloudera.org:8080/#/c/10950/15/be/src/exprs/string-functions.cc@267
PS15, Line 267: VERFLO
> Nit: "Expected"
Done
http://gerrit.cloudera.org:8080/#/c/10950/15/be/src/exprs/string-functions.cc@336
PS15, Line 336: ctx->SetError(msg.c_str());
> line too long (102 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/10950/15/be/src/exprs/string-functions.cc@389
PS15, Line 389: ++i;
> Maybe put ++i on the next line? formatting looks weird.
Done
--
To view, visit http://gerrit.cloudera.org:8080/10950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f
Gerrit-Change-Number: 10950
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 28 Sep 2018 10:42:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7492: Add support for DATE text parser/formatter
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/11450 )
Change subject: IMPALA-7492: Add support for DATE text parser/formatter
..
Patch Set 5: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11450/5/be/src/runtime/date-value.cc
File be/src/runtime/date-value.cc:
http://gerrit.cloudera.org:8080/#/c/11450/5/be/src/runtime/date-value.cc@30
PS5, Line 30: namespace {
nit: Unnamed namespace is not necessary here since the variable is const and
the function is inline => they are already implicitly static
--
To view, visit http://gerrit.cloudera.org:8080/11450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1eec00f22502c4c67c6807c4b51384419ea8b831
Gerrit-Change-Number: 11450
Gerrit-PatchSet: 5
Gerrit-Owner: Attila Jeges
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 28 Sep 2018 10:43:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 9 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 10:54:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/863/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 11:14:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11534 ) Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. IMPALA-7622: adds profile metrics when fetching incremental stats When computing incremental statistics by fetching the stats directly from catalogd, a potentially expensive RPC is made from the impalad coordinator to catalogd. This change adds metrics to the frontend section of the profile to track how long the request takes, the size of the compressed bytes received, and the number of partitions received. The profile for a 'compute incremental ...' command on a table with no statistics looks like this: Frontend: - StatsFetch.CompressedBytes: 0 - StatsFetch.TotalPartitions: 24 - StatsFetch.NumPartitionsWithStats: 0 - StatsFetch.Time: 26ms And the profile looks as follows when the table has stats, so the stats are fetched: Frontend: - StatsFetch.CompressedBytes: 24622 - StatsFetch.TotalPartitions: 23 - StatsFetch.NumPartitionsWithStats: 23 - StatsFetch.Time: 14ms Testing: - manual inspection - e2e test to check the profile Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Reviewed-on: http://gerrit.cloudera.org:8080/11534 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_pull_stats.py 3 files changed, 93 insertions(+), 3 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7622: adds profile metrics when fetching incremental stats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11534 ) Change subject: IMPALA-7622: adds profile metrics when fetching incremental stats .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9b268548c7a98c751eb99855ee08313d1d5a903 Gerrit-Change-Number: 11534 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 11:22:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 16: Code-Review+2 LGTM! Thank you Quanlong for your great work, well done! -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 11:28:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 17 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 11:30:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3243/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 17 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 11:30:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7543: Enhance scan ranges to support sub-ranges
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11520 ) Change subject: IMPALA-7543: Enhance scan ranges to support sub-ranges .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/scan-range.cc File be/src/runtime/io/scan-range.cc: http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/scan-range.cc@268 PS3, Line 268: "There should be no partial reads for sub-ranges." > optional: If this is true, then the loop seems more complex then necessary, Yeah, I think it's not necessary to test eosr in the loop condition, but other than that I don't think it would significantly simplify the code structure. We still have three exit conditions: * buffer is full * we've read all sub-ranges * there was an error during the read Adding more exit branches to the loop body don't simplify the code IMO. And putting them in the for-loop condition doesn't seem natural. Plus, when incrementing sub_range_pos_.index we also need to set sub_range_pos_.bytes_read to zero. -- To view, visit http://gerrit.cloudera.org:8080/11520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea26ba386713990f7671aab5a372cf449b8d51e4 Gerrit-Change-Number: 11520 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 12:00:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7595: Check the validity of the time part of Parquet timestamps
Hello Zoltan Borok-Nagy, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11521 to look at the new patch set (#4). Change subject: IMPALA-7595: Check the validity of the time part of Parquet timestamps .. IMPALA-7595: Check the validity of the time part of Parquet timestamps Before this fix Impala did not check whether a timestamp's time part is out of the valid [0, 24 hour) range when reading Parquet files, so these timestamps were memcopied as they were to slots, leading to results like: 1970-01-01 -00:00:00.1 1970-01-01 24:00:00 Different parts of Impala treat these timestamp differently: - string conversion leads to invalid representation that cannot be converted back to timestamp - timezone conversions handle the overflowing time part and give a valid timestamp result (at least since CCTZ, I did not check older versions of Impala) - Parquet writing inserts these timestamp as they are, so the resulting Parquet file will also contain corrupt timestamps The fix adds a check that converts these corrupt timestamps to NULL, similarly to the handling of timestamp outside the [1400..1) range. A new error code is added for this case. If both the date and the time part is corrupt, then error about corrupt time is returned. Testing: - added a new scanner test that reads a corrupted Parquet file with edge values Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 --- M be/src/exec/parquet-column-readers.cc M be/src/runtime/timestamp-value.h M common/thrift/generate_error_codes.py M testdata/data/README A testdata/data/out_of_range_time_of_day.parquet M testdata/workloads/functional-query/queries/QueryTest/out-of-range-timestamp-abort-on-error.test M testdata/workloads/functional-query/queries/QueryTest/out-of-range-timestamp-continue-on-error.test M tests/query_test/test_scanners.py 8 files changed, 57 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/11521/4 -- To view, visit http://gerrit.cloudera.org:8080/11521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 Gerrit-Change-Number: 11521 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7595: Check the validity of the time part of Parquet timestamps
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11521 ) Change subject: IMPALA-7595: Check the validity of the time part of Parquet timestamps .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/11521/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11521/3//COMMIT_MSG@30 PS3, Line 30: > Yeah, but the current error message is a bit misleading IMO, since currentl I have thought a bit about this and added a new error code. The rationale is that if someone will work on the encoder, then the distinct error codes can help in identifying bugs. http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/exec/parquet-column-readers.cc File be/src/exec/parquet-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/exec/parquet-column-readers.cc@697 PS3, Line 697: !TimestampValue::IsValidDate(val->date()) : || !TimestampValue::IsValidTime(val->time()) > Would it make sense to merge these two functions and have a single Timestam I have kept them as separate functions to be able to distinguish between the two kind of errors. Another reason is that https://gerrit.cloudera.org/#/c/11183/ handles the two errors differently - out of range dates are replaced with not_a_date, while invalid times lead to DCHECK. It is done this way to avoid the performance impact of checking times by relying on callers giving correct values. I plan to check dates with DCHECK too, but that will need some additional changes. http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/runtime/timestamp-value.h File be/src/runtime/timestamp-value.h: http://gerrit.cloudera.org:8080/#/c/11521/3/be/src/runtime/timestamp-value.h@195 PS3, Line 195: than > nit: than Done -- To view, visit http://gerrit.cloudera.org:8080/11521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 Gerrit-Change-Number: 11521 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 13:20:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 9 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 13:21:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3244/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 10 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 13:23:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 10 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 13:23:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7595: Check the validity of the time part of Parquet timestamps
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11521 ) Change subject: IMPALA-7595: Check the validity of the time part of Parquet timestamps .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 Gerrit-Change-Number: 11521 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 13:40:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7492: Add support for DATE text parser/formatter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11450 ) Change subject: IMPALA-7492: Add support for DATE text parser/formatter .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eec00f22502c4c67c6807c4b51384419ea8b831 Gerrit-Change-Number: 11450 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 14:29:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7492: Add support for DATE text parser/formatter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11450 ) Change subject: IMPALA-7492: Add support for DATE text parser/formatter .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3245/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eec00f22502c4c67c6807c4b51384419ea8b831 Gerrit-Change-Number: 11450 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 14:29:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 17: Will this warning fail the GVO? /home/ubuntu/Impala/be/src/exprs/string-functions.cc:142:5: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 17 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 15:05:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3243/ -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 17 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 15:11:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG@15 PS1, Line 15: Made several fixes to exception messages. whats this? http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@97 PS1, Line 97: local_catalog_max_retries Make it more clear, --local_catalog_max_metadata_fetch_retries / --local_catalog_max_plan_retries or something like that? Doesn't tell what is being retried. http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@99 PS1, Line 99: frontend Just to be sure, do we plan to skip these retries for non-fe paths? Like directly accessing the Catalog http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py File tests/custom_cluster/test_local_catalog.py: http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@220 PS1, Line 220: Tests that the flag to limit the number of retries works Maybe I'm missing something but we don't seem to be doing this? We set max_retries=0 and asserting total_ex_throws > 0. http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@251 PS1, Line 251: else, do we need to re-throw and fail? -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 15:28:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7595: Check the validity of the time part of Parquet timestamps
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11521 ) Change subject: IMPALA-7595: Check the validity of the time part of Parquet timestamps .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/11521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 Gerrit-Change-Number: 11521 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 16:07:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11539 to look at the new patch set (#2). Change subject: IMPALA-7599: make the number of local cache retries configurable .. IMPALA-7599: make the number of local cache retries configurable Under heavy read/write load, the number of retries needed for queries in order to skip over inconsistent metadata exceptions needs to be set higher. This change makes the number of retries configurable. It can be set with the newly added flag --local_catalog_max_retries. In addition, this change increases the default from 10 to 40, which was sufficient when handling several workloads with high read/write load. Made several fixes to exception messages. Testing: - manual tests - added an e2e test that sets the flag and checks for inconsistent metadata Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_local_catalog.py 8 files changed, 71 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/2 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG@15
PS1, Line 15: Made several fixes to exception messages.
> whats this?
whoops, added to a wrong branch... thanks for catching it.
when invalidate exception is thrown, the message was messed up due to {}
instead of %s. the "name" of the object did not make sense either and was
inconsistent with the other case where there this exception is thrown.
the change is in CatalogMetaProvider.
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@97
PS1, Line 97: local_catalog_max_fetch_r
> Make it more clear, --local_catalog_max_metadata_fetch_retries / --local_ca
added 'fetch'. metadata and catalog in the same name seems redundant.
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@99
PS1, Line 99: frontend
> Just to be sure, do we plan to skip these retries for non-fe paths? Like di
clarified that its use is for accessing the local catalog's objects.
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@220
PS1, Line 220: for c in [client1, client2]]
> Maybe I'm missing something but we don't seem to be doing this? We set max_
tightened this to check that we never retry (since the retries are set to 0).
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@251
PS1, Line 251:
> else, do we need to re-throw and fail?
the counter will be incorrect, so it'll fail. I'm not excluding other
exceptions here.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 28 Sep 2018 16:49:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11539 to look at the new patch set (#3). Change subject: IMPALA-7599: make the number of local cache retries configurable .. IMPALA-7599: make the number of local cache retries configurable Under heavy read/write load, the number of retries needed for queries in order to skip over inconsistent metadata exceptions needs to be set higher. This change makes the number of retries configurable. It can be set with the newly added flag --local_catalog_max_fetch_retries. In addition, this change increases the default from 10 to 40, which was sufficient when handling several workloads with high read/write load. Made several fixes to exception messages. Testing: - manual tests - added an e2e test that sets the flag and checks for inconsistent metadata Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_local_catalog.py 8 files changed, 71 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/3 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 10 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 17:05:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7503: SHOW GRANT USER not showing all privileges.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11531 ) Change subject: IMPALA-7503: SHOW GRANT USER not showing all privileges. .. IMPALA-7503: SHOW GRANT USER not showing all privileges. This patch fixes the SHOW GRANT USER statement to show all privileges granted to a user, either directly via object ownership, or granted through a role via a group the user belongs to. The output for SHOW GRANT USER will have two additional columns for privilege name and privilege type so the user can know where the privilege comes from. Truncated sample showing two columns that are different from role: ++++--+-... | principal_type | principal_name | scope | database | ... ++++--+-... | USER | foo| table | foo_db | ... | ROLE | foo_role | server | | ... ++++--+-... Testing: - Create new custom cluster test with custom group mapping. - Ran FE and custom cluster tests. Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Reviewed-on: http://gerrit.cloudera.org:8080/11531 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/catalog/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java A fe/src/test/java/org/apache/impala/service/CustomClusterGroupMapper.java A fe/src/test/java/org/apache/impala/service/CustomClusterResourceAuthorizationProvider.java M fe/src/test/resources/sentry-site_no_oo.xml.template M fe/src/test/resources/sentry-site_oo.xml.template M fe/src/test/resources/sentry-site_oo_nogrant.xml.template M testdata/bin/run-sentry-service.sh A testdata/workloads/functional-query/queries/QueryTest/show_grant_user.test M tests/authorization/test_grant_revoke.py M tests/authorization/test_owner_privileges.py A tests/authorization/test_show_grant_user.py M tests/common/sentry_cache_test_suite.py 15 files changed, 1,400 insertions(+), 177 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie9f6c88f5569e1c414ceb8a86e7b013eaa3ecde1 Gerrit-Change-Number: 11531 Gerrit-PatchSet: 11 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/865/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 17:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7492: Add support for DATE text parser/formatter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11450 ) Change subject: IMPALA-7492: Add support for DATE text parser/formatter .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eec00f22502c4c67c6807c4b51384419ea8b831 Gerrit-Change-Number: 11450 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 18:14:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7492: Add support for DATE text parser/formatter
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11450 ) Change subject: IMPALA-7492: Add support for DATE text parser/formatter .. IMPALA-7492: Add support for DATE text parser/formatter This change is the first step in implementing support for DATE type (IMPALA-6169). The DATE parser/formatter is implemented by the new DateParser class. - The parser supports parsing both default and custom formatted DATE values. CCTZ is used to validate the parsed dates. - The formatter supports default and custom formatting of DATE values. In the future, DateParser will be used in the text scanner/writer and in the DATE <-> STRING cast functions. The DateParser class reuses some of the functionality already implemented in the TimestampParser class to minimize redundancy. To make code reuse easier, a new namespace (datetime_parse_util) was created and the common functionality was moved there. This change also adds a new class (DateValue) to represent a DATE value in-memory. The DateParser and DateValue classes are used only in tests at the moment, therefore this patch doesn't change user facing behavior. Testing: - Added BE-tests for DateParser and DateValue classes. - Re-run parse-timestamp-benchmark to make sure that parser performance hasn't degraded. Change-Id: I1eec00f22502c4c67c6807c4b51384419ea8b831 Reviewed-on: http://gerrit.cloudera.org:8080/11450 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/benchmarks/parse-timestamp-benchmark.cc M be/src/common/init.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc M be/src/exprs/timestamp-functions.h M be/src/runtime/CMakeLists.txt A be/src/runtime/date-parse-util.cc A be/src/runtime/date-parse-util.h A be/src/runtime/date-test.cc A be/src/runtime/date-value.cc A be/src/runtime/date-value.h A be/src/runtime/datetime-parse-util.cc A be/src/runtime/datetime-parse-util.h M be/src/runtime/timestamp-parse-util.cc M be/src/runtime/timestamp-parse-util.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h 19 files changed, 1,896 insertions(+), 732 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1eec00f22502c4c67c6807c4b51384419ea8b831 Gerrit-Change-Number: 11450 Gerrit-PatchSet: 7 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect
anujphadke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11540 Change subject: [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect .. [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect impala-shell does not set any socket timeout while connecting to the impala server. This change sets a timeout on the socket before connecting and unsets it back after successfully connecting. The default timeout on this socket is 1 min. Usage: impala-shell --client_connect_timeout= Testing: 1. Added a test where I create a random listening socket. impala-shell connects to this socket and times out after the default timeout value. 2. Created a kerberized impala cluster with ssl enabled and connected to the impalad using an openssl client (block the beeswax server thread to accept new connection) - E.g. - openssl s_client -connect :21000 Used impala-shell to connect to the same impalad later. impala-shell timed out after the default of 1 min.I verified it manually. Change-Id: I130fc47f7a83f591918d6842634b4e5787d00813 --- M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/shell/test_shell_commandline.py 5 files changed, 33 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/11540/1 -- To view, visit http://gerrit.cloudera.org:8080/11540 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I130fc47f7a83f591918d6842634b4e5787d00813 Gerrit-Change-Number: 11540 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke
[Impala-ASF-CR] [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11540 ) Change subject: [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/11540/1/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/11540/1/shell/option_parser.py@226 PS1, Line 226: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@744 PS1, Line 744: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@747 PS1, Line 747: r flake8: E501 line too long (94 > 90 characters) http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@748 PS1, Line 748: r flake8: F841 local variable 'results' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/11540 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I130fc47f7a83f591918d6842634b4e5787d00813 Gerrit-Change-Number: 11540 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 18:45:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 17: Looks like a clang-tidy failure: 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:142:5: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:144:5: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:218:3: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:243:3: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:277:3: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:285:3: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:326:3: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:327:3: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] 12:13:38 /home/ubuntu/Impala/be/src/exprs/string-functions.cc:367:9: warning: unused exception parameter 'e' [clang-diagnostic-unused-exception-parameter] -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 17 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 19:01:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7595: Check the validity of the time part of Parquet timestamps
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11521 ) Change subject: IMPALA-7595: Check the validity of the time part of Parquet timestamps .. Patch Set 4: Code-Review+2 (1 comment) Thank you for fixing Csaba!! http://gerrit.cloudera.org:8080/#/c/11521/4/common/thrift/generate_error_codes.py File common/thrift/generate_error_codes.py: http://gerrit.cloudera.org:8080/#/c/11521/4/common/thrift/generate_error_codes.py@370 PS4, Line 370:"The time of day should be 0 <= and < 24 hour (in nanoseconds)."), Worth including the bogus value? -- To view, visit http://gerrit.cloudera.org:8080/11521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc0ae651b6a0a028c61a15fd069ef9e904231058 Gerrit-Change-Number: 11521 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 19:03:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11540 ) Change subject: [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/866/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11540 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I130fc47f7a83f591918d6842634b4e5787d00813 Gerrit-Change-Number: 11540 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 19:16:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Hello Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11528 to look at the new patch set (#7). Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. IMPALA-7310: All-null columns give wrong estimates in planner Modified the planner to handle low-value NDVs by adjusting them upward by one to account for null values. Thus, even an all-null column, which has an NDV of 0 in stats, will have an NDV of 1 in the planner. (The planner already expects NDV to include nulls.) Modified the front end to allow capturing the full plan for use in a unit test. Added unit tests that verify estimated cardinality for a plan as a way to verify that the fix solved the scenario in IMPALA-7310. Testing required a new table, similar to the existing nulltable, but which has multiple rows and has stats calculated. The change was limited to a very narrow range of cases: * Table column (not an internal column such as COUNT(*)) * Column is nullable * Column has stats * Column does not provide a null count, or null count > 0 * Reported NDV <= 1 In this narrow case, we add one to NDV to account for nulls. (Any larger adjustment throws off the TPC-H tests which have multiple columns, marked as non-null, with low NDV, but which actually include no nulls.) The change minimized impact on PlannerTest, but still some memory numbers needed adjusting for a test in which one column hit the criteria listed above and had its NDV adjusted. Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/ExprNdvTest.java A fe/src/test/java/org/apache/impala/planner/CardinalityTest.java A testdata/NullTable/large_data.csv M testdata/bin/compute-table-stats.sh M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test 11 files changed, 443 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/11528/7 -- To view, visit http://gerrit.cloudera.org:8080/11528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Gerrit-Change-Number: 11528 Gerrit-PatchSet: 7 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/11528 )
Change subject: IMPALA-7310: All-null columns give wrong estimates in planner
..
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11528/7/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/11528/7/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@111
PS7, Line 111: expectCardinality("SELECT d FROM functional.alltypes,
functional.nullrows", 7300 * 26);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11528/7/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@143
PS7, Line 143: expectCardinality("SELECT COUNT(*)" + joinClause + "GROUP BY
t1.id, t1.int_col", 7300 * 10);
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/11528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459
Gerrit-Change-Number: 11528
Gerrit-PatchSet: 7
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Fri, 28 Sep 2018 19:41:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Hello Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11528 to look at the new patch set (#8). Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. IMPALA-7310: All-null columns give wrong estimates in planner Modified the planner to handle low-value NDVs by adjusting them upward by one to account for null values. Thus, even an all-null column, which has an NDV of 0 in stats, will have an NDV of 1 in the planner. (The planner already expects NDV to include nulls.) Modified the front end to allow capturing the full plan for use in a unit test. Added unit tests that verify estimated cardinality for a plan as a way to verify that the fix solved the scenario in IMPALA-7310. Testing required a new table, similar to the existing nulltable, but which has multiple rows and has stats calculated. The change was limited to a very narrow range of cases: * Table column (not an internal column such as COUNT(*)) * Column is nullable * Column has stats * Column does not provide a null count, or null count > 0 * Reported NDV <= 1 In this narrow case, we add one to NDV to account for nulls. (Any larger adjustment throws off the TPC-H tests which have multiple columns, marked as non-null, with low NDV, but which actually include no nulls.) The change minimized impact on PlannerTest, but still some memory numbers needed adjusting for a test in which one column hit the criteria listed above and had its NDV adjusted. Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/ExprNdvTest.java A fe/src/test/java/org/apache/impala/planner/CardinalityTest.java A testdata/NullTable/large_data.csv M testdata/bin/compute-table-stats.sh M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test 11 files changed, 455 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/11528/8 -- To view, visit http://gerrit.cloudera.org:8080/11528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Gerrit-Change-Number: 11528 Gerrit-PatchSet: 8 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11528 ) Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/867/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Gerrit-Change-Number: 11528 Gerrit-PatchSet: 7 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 28 Sep 2018 20:16:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 18 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 28 Sep 2018 20:18:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 17 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 28 Sep 2018 20:18:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3246/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 18 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 28 Sep 2018 20:18:40 + Gerrit-HasComments: No
[Impala-ASF-CR] Minor cleanup of hash table probe counts
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11532 ) Change subject: Minor cleanup of hash table probe counts .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I179d42300b069d0a34da30bb593d8f97b5846dc8 Gerrit-Change-Number: 11532 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 28 Sep 2018 20:22:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11528 ) Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/868/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Gerrit-Change-Number: 11528 Gerrit-PatchSet: 8 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 28 Sep 2018 20:42:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Minor cleanup of hash table probe counts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11532 ) Change subject: Minor cleanup of hash table probe counts .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I179d42300b069d0a34da30bb593d8f97b5846dc8 Gerrit-Change-Number: 11532 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 28 Sep 2018 21:01:10 + Gerrit-HasComments: No
[Impala-ASF-CR] Minor cleanup of hash table probe counts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11532 ) Change subject: Minor cleanup of hash table probe counts .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3247/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I179d42300b069d0a34da30bb593d8f97b5846dc8 Gerrit-Change-Number: 11532 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 28 Sep 2018 21:01:11 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11540 ) Change subject: [WIP] IMPALA-7555: Impala-shell should timeout if it fails to connect .. Patch Set 1: (17 comments) http://gerrit.cloudera.org:8080/#/c/11540/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11540/1//COMMIT_MSG@7 PS1, Line 7: should Rephrase to state what the change does, not what it should do, e.g. "Set socket timeout in impala-shell" http://gerrit.cloudera.org:8080/#/c/11540/1//COMMIT_MSG@19 PS1, Line 19: 2. Created a kerberized impala cluster with ssl enabled and connected This should be automated, too. http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py@81 PS1, Line 81: client_connect_timeout Is this milliseconds? If so, please rename to client_connect_timeout_ms http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py@254 PS1, Line 254: ret = self._get_socket_and_transport() : sock, self.transport = ret[0], ret[1] You can write this in a single line: sock, self.transport = self._get_socket_and_transport() http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py@257 PS1, Line 257: self.transport.open() : protocol = TBinaryProtocol.TBinaryProtocol(self.transport) : self.imp_service = ImpalaService.Client(protocol) : result = self.ping_impala_service() Which of these 4 lines can time out? Which of the ones that can time out do we currently test? There doesn't seem to be a test that connects successfully but times out in the ssl connection setup. http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py@261 PS1, Line 261: sock.setTimeout(None) This could be more clear if we pass 0 instead http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py@274 PS1, Line 274: """Create a Transport. update the comment http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_client.py@326 PS1, Line 326: return TSaslClientTransport(sasl_factory, "GSSAPI", sock) This will now return a tuple in some cases, and a transport in others. Please make it consistent so that the function returns the same types in all cases. http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_shell.py@864 PS1, Line 864: self.imp_client.close_connection() Please add a test that calls connect from the shell instead of passing a host during startup http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_shell_config_defaults.py File shell/impala_shell_config_defaults.py: http://gerrit.cloudera.org:8080/#/c/11540/1/shell/impala_shell_config_defaults.py@54 PS1, Line 54: 'client_connect_timeout': 6, How did you decide on 60s? Why not pick something smaller, e.g. 5s? http://gerrit.cloudera.org:8080/#/c/11540/1/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/11540/1/shell/option_parser.py@225 PS1, Line 225: client_connect_timeout client_connect_timeout_ms? http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@737 PS1, Line 737: Test nit: Tests http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@743 PS1, Line 743: s = socket.socket() Use contextlib.closing here: https://docs.python.org/2/library/contextlib.html#contextlib.closing http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@745 PS1, Line 745: 1 Please add a comment what this 1 does? http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@746 PS1, Line 746: impalad test_port? It's not an impalad http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@747 PS1, Line 747: r > flake8: E501 line too long (94 > 90 characters) use ' and " consistently in this line http://gerrit.cloudera.org:8080/#/c/11540/1/tests/shell/test_shell_commandline.py@747 PS1, Line 747: 6 This means that the test takes 60s to execute, right? Can you pick a smaller value instead, e.g. 2s? -- To view, visit http://gerrit.cloudera.org:8080/11540 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I130fc47f7a83f591918d6842634b4e5787d00813 Gerrit-Change-Number: 11540 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 28 Sep 2018 21:21:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/11528 )
Change subject: IMPALA-7310: All-null columns give wrong estimates in planner
..
Patch Set 8:
(10 comments)
first pass comments. will make another pass.
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@24
PS8, Line 24: * Table column (not an internal column such as COUNT(*))
Base table?
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@27
PS8, Line 27: not provide a null count
when does this happen when stats are computed?
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@28
PS8, Line 28: NDV <= 1
if its unknown whether there are nulls or not (from the prev. line), could this
over count?
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@31
PS8, Line 31: Any larger adjustment
confused... if we're adjusting for nulls, and we're treating a null as a single
value, why is a larger adjustment than +1 considered?
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@111
PS8, Line 111: public ArrayList getGroupList() { return groupingExprs_;
}
where is this used in this change?
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/main/java/org/apache/impala/analysis/SlotRef.java
File fe/src/main/java/org/apache/impala/analysis/SlotRef.java:
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/main/java/org/apache/impala/analysis/SlotRef.java@41
PS8, Line 41: add value
nit: lots of 'value' in this comment. perhaps replace this with "make sense".
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@22
PS8, Line 22: private static final boolean DEBUG_MODE = false;
use a logger.
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@190
PS8, Line 190: if (DEBUG_MODE) {
: System.out.println(plan.get(0).getExplainString(
: queryOptions, TExplainLevel.EXTENDED));
: }
use a logger
http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/NullTable/large_data.csv
File testdata/NullTable/large_data.csv:
http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/NullTable/large_data.csv@1
PS8, Line 1: a,,\N,\N,\N,a
why's this called "large_data"?
http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/datasets/functional/functional_schema_template.sql@1355
PS8, Line 1355: CREATE_KUDU
where's this used for kudu tests? fwict, the refs to this new table are
explicitly through java unit tests so I don't see anything that uses the kudu
instance
--
To view, visit http://gerrit.cloudera.org:8080/11528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459
Gerrit-Change-Number: 11528
Gerrit-PatchSet: 8
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 28 Sep 2018 21:55:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7532: Add catalogd client backoff time into impalad CLI options
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11543 Change subject: IMPALA-7532: Add catalogd client backoff time into impalad CLI options .. IMPALA-7532: Add catalogd client backoff time into impalad CLI options Impala may fail queries or fail to start if the connection to catalogd cannot be estabilished. Impala already has a retrial mechanism but the backoff time is currently 0. This patch adds an option for it, defaulting to 10 seconds. Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 --- M be/src/runtime/exec-env.cc 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/11543/2 -- To view, visit http://gerrit.cloudera.org:8080/11543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 Gerrit-Change-Number: 11543 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
..
Patch Set 3: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG@16
PS3, Line 16:
May be mention that all other retries (outside of Frontend) will come in a
separate patch?
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1090
PS3, Line 1090: timeline.markEvent(
: String.format("Retrying query planning due to
inconsistent metadata "
: + "fetch, attempt %s of %s: ",
: attempt, INCONSISTENT_METADATA_NUM_RETRIES)
While we are here, can we move this out of the loop? messes up run-time profile
if logged for 40 times. May be we include the number of retries it took for it
to be successful?
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 28 Sep 2018 23:17:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Hello Greg Rahn, Zoltan Borok-Nagy, Attila Jeges, Todd Lipcon, Tim Armstrong,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10950
to look at the new patch set (#18).
Change subject: IMPALA-376: add built-in functions for parsing JSON
..
IMPALA-376: add built-in functions for parsing JSON
This patch implements the same function as Hive UDF get_json_object.
We reuse RapidJson to parse the json string. In order to track the
memory used in RapidJson, we wrap FunctionContext into an allocator.
get_json_object accepts two parameters: a json string and a selector
(json path). We parse the json string into a Document tree and then
perform BFS according to the selector. For example, to process
get_json_object('[{\"a\":1}, {\"a\":2}, {\"a\":3}]', '$[*].a'),
we first perform '$[*]' to extract all the items in the root array.
Then we get a queue consists of {a:1},{a:2},{a:3} and perform '.a'
selector on all values in the queue. The final results is 1,2,3 in the
queue. As there're multiple results, they should be encapsulated into
an array. The output results is a string of '[1,2,3]'.
More examples can be found in expr-test.cc.
Test:
* Add unit tests in expr-test
* Add e2e tests in exprs.test
* Add tests in test_alloc_fail.py to check handling of out of memory
Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f
---
M be/src/exprs/CMakeLists.txt
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
A be/src/exprs/string-functions.cc
M be/src/exprs/string-functions.h
M be/src/util/string-util.cc
M be/src/util/string-util.h
M common/function-registry/impala_functions.py
M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-init.test
M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
11 files changed, 658 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/10950/18
--
To view, visit http://gerrit.cloudera.org:8080/10950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f
Gerrit-Change-Number: 10950
Gerrit-PatchSet: 18
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7532: Add catalogd client backoff time into impalad CLI options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11543 ) Change subject: IMPALA-7532: Add catalogd client backoff time into impalad CLI options .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/869/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 Gerrit-Change-Number: 11543 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 28 Sep 2018 23:24:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Paul Rogers has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11544 Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. IMPALA-7310: All-null columns give wrong estimates in planner Modified the planner to handle low-value NDVs by adjusting them upward by one to account for null values. Thus, even an all-null column, which has an NDV of 0 in stats, will have an NDV of 1 in the planner. (The planner already expects NDV to include nulls.) Modified the front end to allow capturing the full plan for use in a unit test. Added unit tests that verify estimated cardinality for a plan as a way to verify that the fix solved the scenario in IMPALA-7310. Testing required a new table, similar to the existing nulltable, but which has multiple rows and has stats calculated. The change was limited to a very narrow range of cases: * Base table column (not an internal column such as COUNT(*)) * Type is not BOOLEAN (turns out metadata does the needed NDV correction for BOOLEAN only.) * Column has stats * Column is nullable * Column does not provide a null count, or null count > 0 * Reported NDV <= 1 Testing showed that, at least for the functional test tables, we do have cases in which stats are computed, but the null count is -1 (undefined), which is why null count had to be considered. If we know the null count, and the null count is zero, then no adjustment is needed, But, if we don't know the null count, or it is positive, then adjustment may be needed. Research for this patch revealed that Impala treats NDVs in two distinct ways: * Stats (which use the NDV function) computes NDV as the number of distinct non-null values. (That is, the NDV of (0, null) is 1.) * The planner itself when working with constants, uses a definition of NDV that includes nulls. That is, the NDV of (0, null) is 2. This fix attempts to bridge the two definitions, Since we know that the NDV in stats excludes nulls (except for the BOOLEAN type), and we know that the column contains nulls, we can bump up the NDV to convert from the stats definition to the planner definition. But, to avoid regressions, we do so in a very narrow range of NDV values: only 0 and 1. Technically, the adjustment should apply to all NDV values. However, it turns out that if we do so, we end up with many failures in PlannerTest in those tests that work with TPC-H tables. The TPC-H tables have multiple columns marked as nullable but which actually have no nulls. Some of these columns also have a low NDV. By limiting the NDV adjustment to the narrow range, the TPC-H tests need not be updated. Since end users could have a similar situation, the narrow range reduces the chance that this fix might impact such workloads. Although the change minimized impact on PlannerTest, some memory numbers needed adjusting for a test in which one column hit the criteria listed above and had its NDV adjusted. Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Fixes Change-Id: I282951f20598839fad880995a032e016845083db --- M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/ExprNdvTest.java A fe/src/test/java/org/apache/impala/planner/CardinalityTest.java A testdata/NullTable/large_data.csv M testdata/bin/compute-table-stats.sh M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test 10 files changed, 443 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/11544/1 -- To view, visit http://gerrit.cloudera.org:8080/11544 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I282951f20598839fad880995a032e016845083db Gerrit-Change-Number: 11544 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Paul Rogers has posted comments on this change. (
http://gerrit.cloudera.org:8080/11528 )
Change subject: IMPALA-7310: All-null columns give wrong estimates in planner
..
Patch Set 8:
(32 comments)
Thanks for the reviews. Here are my responses thus far. Most are of the form
"did it." Vuk, I had a couple of questions on your comments.
Thanks,
- Paul
http://gerrit.cloudera.org:8080/#/c/11528/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11528/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7310: All-null columns give wrong estimates in planner
> We tend to have a single line subject with a newline after it. That ends up
Other projects require the JIRA ticket number in the commit message. Do we?
Such projects tend to favor the JIRA ticket title (or a cleaned up version) as
the description. Do we? In these projects, the assumption is that the patch is
to fix the listed ticket, so no need to include the word "Fix" in the patch
title.
Happy to remove the ticket number and change the title to "Correct NDV
estimates for all-null columns."
I pushed an update with the newline which shows up in my view of the patch.
Did that revision replace the one here on which you commented? Is there a
better way to revise the commit message?
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@24
PS8, Line 24: * Table column (not an internal column such as COUNT(*))
> Base table?
Ack
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@27
PS8, Line 27: not provide a null count
> when does this happen when stats are computed?
See revised message. I saw multiple cases in the test DBs in which stats were
computed, but null count was -1 (undefined), so I needed to handle that case in
order to get things to work.
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@28
PS8, Line 28: NDV <= 1
> if its unknown whether there are nulls or not (from the prev. line), could
Added to the commit message:
Research for this patch revealed that Impala treats NDVs in two distinct ways:
* Stats (which use the NDV function) computes NDV as the number of distinct
non-null values. (That is, the NDV of (0, null) is 1.
* The planner itself when working with constants, uses a definition of NDV that
includes nulls. That is, the NDV of (0, null) is 2.
This fix attempts to bridge the two definitions, Since we know that the NDV in
stats excludes nulls (except for the BOOLEAN type), and we know that the column
contains nulls, we can bump up the NDV to convert from the stats definition to
the planner definition. But, to avoid regressions, we do so in a very narrow
range of NDV values: only 0 and 1.
http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@31
PS8, Line 31: Any larger adjustment
> confused... if we're adjusting for nulls, and we're treating a null as a si
Reworded:
Technically, the adjustment should apply to all NDV values. However, it turns
out that if we do so, we end up with many failures in PlannerTest in those
tests that work with TPC-H tables. The TPC-H tables have multiple columns
marked as nullable but which actually have no nulls. Some of these columns also
have a low NDV.
By limiting the NDV adjustment to the narrow range, the TPC-H tests need not be
updated. Since end users could have a similar situation, the narrow range
reduces the chance that this fix might impact such workloads.
http://gerrit.cloudera.org:8080/#/c/11528/2/.gitignore
File .gitignore:
http://gerrit.cloudera.org:8080/#/c/11528/2/.gitignore@48
PS2, Line 48:
> I'm not a stickler, but you could separate the .gitignore stuff into a sepa
Reverted all changes to this file.
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/11528/8/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@111
PS8, Line 111: public ArrayList getGroupList() { return groupingExprs_;
}
> where is this used in this change?
Was used in a test, but that test ended up not being include as part of this
patch, so removed the line.
http://gerrit.cloudera.org:8080/#/c/11528/2/fe/src/main/java/org/apache/impala/analysis/SlotRef.java
File fe/src/main/java/org/apache/impala/analysis/SlotRef.java:
http://gerrit.cloudera.org:8080/#/c/11528/2/fe/src/main/java/org/apache/impala/analysis/SlotRef.java@102
PS2, Line 102: private void computeNdv() {
> line has trailing whitespace
Ack
http://gerrit.cloudera.org:8080/#/c/11528/2/fe/src/main/java/org/apache/impala/analysis/SlotRef.java@105
PS2, Line 105: // Potentially adjust NDV for nulls if the column is has
stats,
> This used to execute even when hasStats() was false. What's numDistinctValu
Defaults to -1.
Put it back as it
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG@16
PS3, Line 16:
> May be mention that all other retries (outside of Frontend) will come in a
Done. Note that most of these are from Frontend as well, but there are several
cases outside as well.
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1090
PS3, Line 1090: timeline.markEvent(
: String.format("Retrying query planning due to
inconsistent metadata "
: + "fetch, attempt %s of %s: ",
: attempt, INCONSISTENT_METADATA_NUM_RETRIES)
> While we are here, can we move this out of the loop? messes up run-time pro
avoided the repetition by printing a summary to the profile. note that it'll
only be the last exception that's printed... conceivably, we may fail for a
variety of reasons but I think its ok to get this level of detail from the logs.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 28 Sep 2018 23:42:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11539 to look at the new patch set (#4). Change subject: IMPALA-7599: make the number of local cache retries configurable .. IMPALA-7599: make the number of local cache retries configurable Under heavy read/write load, the number of retries needed for queries in order to skip over inconsistent metadata exceptions needs to be set higher. This change makes the number of retries configurable. It can be set with the newly added flag --local_catalog_max_fetch_retries. In addition, this change increases the default from 10 to 40, which was sufficient when handling several workloads with high read/write load. Follow-up change for IMPALA-7597 will make use of this configuration when retrying for cases other than analyzing queries. Made several fixes to exception messages. Testing: - manual tests - added an e2e test that sets the flag and checks for inconsistent metadata Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_local_catalog.py 8 files changed, 87 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/4 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/870/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 18 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 28 Sep 2018 23:48:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11544 ) Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/871/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11544 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I282951f20598839fad880995a032e016845083db Gerrit-Change-Number: 11544 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Sep 2018 23:54:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/872/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 00:00:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 18: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 18 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Sat, 29 Sep 2018 00:08:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. IMPALA-6271: Impala daemon should log a message when it's being shut down Currently Impalad does not log any message when SIGTERM is sent to impalad to terminate or to do a graceful shut down. This change logs a message when SIGTERM is received by impalad/catalogd/statestored. This logging will assist in debugging the issues seen in the field where impalad was not gracefully shut down (some other signal was generated that led to impalad/catalogd/statestored crash). Testing: --- a) Used kill to send signals to impalad/catalogd/statestored `kill -s SIGTERM ` and see the log message is being logged in impalad/catalogd/statestored.INFO. b) Ran test_breakpad.py to check that existing breakpad functionalities are not affected. c) Ran exhaustive tests without failure. d) Added new test in test_breakpad.py to handle SIGTERM for impalad/statestored/catalogd. Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Reviewed-on: http://gerrit.cloudera.org:8080/10847 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/catalog/catalogd-main.cc M be/src/common/init.cc M be/src/util/minidump.cc M tests/custom_cluster/test_breakpad.py 4 files changed, 64 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 19 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7310: All-null columns give wrong estimates in planner
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11528 ) Change subject: IMPALA-7310: All-null columns give wrong estimates in planner .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11528/8//COMMIT_MSG@31 PS8, Line 31: Any larger adjustment > Reworded: what types of changes are these-- did they manage to change the structure of the plans? http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/NullTable/large_data.csv File testdata/NullTable/large_data.csv: http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/NullTable/large_data.csv@1 PS8, Line 1: a,,\N,\N,\N,a > Because the directory already contains a "data.csv" file. Couldn't justify Looks like theres several conventions under testdata... I see testdata/data has some files that look ad hoc. perhaps there? http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/11528/8/testdata/datasets/functional/functional_schema_template.sql@1355 PS8, Line 1355: CREATE_KUDU > Sorry, just copied the one-line version of this file. Can't claim I underst I have no idea, but I counted BASE_TABLE_NAME and see it shows up 105 times whereas the CREATE_KUDU string shows up 16 times. So it looks like we do this selectively as test cases require. -- To view, visit http://gerrit.cloudera.org:8080/11528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Gerrit-Change-Number: 11528 Gerrit-PatchSet: 8 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 00:20:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-376: add built-in functions for parsing JSON
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10950 ) Change subject: IMPALA-376: add built-in functions for parsing JSON .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9d3598cb3beca0865a7edb094f3a5b602dbd2f Gerrit-Change-Number: 10950 Gerrit-PatchSet: 18 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 29 Sep 2018 00:31:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11539 to look at the new patch set (#5). Change subject: IMPALA-7599: make the number of local cache retries configurable .. IMPALA-7599: make the number of local cache retries configurable Under heavy read/write load, the number of retries needed for queries in order to skip over inconsistent metadata exceptions needs to be set higher. This change makes the number of retries configurable. It can be set with the newly added flag --local_catalog_max_fetch_retries. In addition, this change increases the default from 10 to 40, which was sufficient when handling several workloads with high read/write load. Follow-up change for IMPALA-7597 will make use of this configuration when retrying for cases other than analyzing queries. Made several fixes to exception messages. Testing: - manual tests - added an e2e test that sets the flag and checks for inconsistent metadata Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_local_catalog.py 8 files changed, 93 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/5 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 5: Code-Review+2 carry +2 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 00:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 6 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 00:42:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3248/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 6 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 00:42:09 + Gerrit-HasComments: No
[Impala-ASF-CR] Minor cleanup of hash table probe counts
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11532 ) Change subject: Minor cleanup of hash table probe counts .. Minor cleanup of hash table probe counts * Increment num_probes_ in Probe() instead of in every caller * Removed dead num_failed_probes_ variable. Initialise some members to constants inline while we're here. Change-Id: I179d42300b069d0a34da30bb593d8f97b5846dc8 Reviewed-on: http://gerrit.cloudera.org:8080/11532 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/hash-table.inline.h 3 files changed, 18 insertions(+), 36 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I179d42300b069d0a34da30bb593d8f97b5846dc8 Gerrit-Change-Number: 11532 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] Minor cleanup of hash table probe counts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11532 ) Change subject: Minor cleanup of hash table probe counts .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I179d42300b069d0a34da30bb593d8f97b5846dc8 Gerrit-Change-Number: 11532 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Sat, 29 Sep 2018 00:43:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7532: Add catalogd client backoff time into impalad CLI options
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11543 ) Change subject: IMPALA-7532: Add catalogd client backoff time into impalad CLI options .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/11543/2/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/11543/2/be/src/runtime/exec-env.cc@128 PS2, Line 128: catalog_client_rpc_backoff_ms I see there's a rpc_retry_interval_ms flag. For consistency, should this be named catalog_client_rpc_retry_internal_ms? http://gerrit.cloudera.org:8080/#/c/11543/2/be/src/runtime/exec-env.cc@129 PS2, Line 129: retrial nit: retrying -- To view, visit http://gerrit.cloudera.org:8080/11543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I924c1f2fd37021f4c8fb6b46aa278ac4b1aee131 Gerrit-Change-Number: 11543 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 00:56:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/873/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 01:27:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 6 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 29 Sep 2018 04:41:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11539 ) Change subject: IMPALA-7599: make the number of local cache retries configurable .. IMPALA-7599: make the number of local cache retries configurable Under heavy read/write load, the number of retries needed for queries in order to skip over inconsistent metadata exceptions needs to be set higher. This change makes the number of retries configurable. It can be set with the newly added flag --local_catalog_max_fetch_retries. In addition, this change increases the default from 10 to 40, which was sufficient when handling several workloads with high read/write load. Follow-up change for IMPALA-7597 will make use of this configuration when retrying for cases other than analyzing queries. Made several fixes to exception messages. Testing: - manual tests - added an e2e test that sets the flag and checks for inconsistent metadata Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Reviewed-on: http://gerrit.cloudera.org:8080/11539 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_local_catalog.py 8 files changed, 93 insertions(+), 12 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8 Gerrit-Change-Number: 11539 Gerrit-PatchSet: 7 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
